Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data

ABSTRACT

The invention facilitates and enhances review of a body of information (that can be represented by a set of audio data, video data, text data or some combination of the three), enabling the body of information to be quickly reviewed to obtain an overview of the content of the body of information and allowing flexibility in the manner in which the body of information is reviewed. In a particular application of the invention, the content of audiovisual news programs is acquired from a first set of one or more information sources (e.g., television news programs) and text news stories are acquired from a second set of one or more information sources (e.g., on-line news services or news wire services). In such a particular application, the invention can enable the user to access the news stories of audiovisual news programs in a random manner so that the user can move quickly among news stories or news programs. The invention can also enable the user to quickly locate news stories pertaining to a particular subject. Additionally, when the user is observing a particular news story in a news program, the invention can identify and display related news stories. The invention can also enable the user to control the display of the news programs by, for example, speeding up the display, causing a summary of one or more news stories to be displayed, or pausing the display of the news stories. Additionally, the invention can indicate to the user which news story is currently being viewed, as well as which news stories have previously been viewed.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to systems and methods that enable observation ofa body of information and, in particular, a body of information that canbe represented, at least in part, by audiovisual data. Mostparticularly, the invention relates to systems and methods for accessingand reviewing a body of information represented by one or more sets ofaudiovisual data that can be used to generate an audiovisual display andone or more related sets of text data that can be used to generate atext display.

2. Related Art

The increasing complexity of the modern world, and the concomitantexplosion in the amount of information available to describe that world,has placed competing demands on people. There is more subject matterthat people find necessary or desirable to master or, at least, befamiliar with. At the same time, there is less time to spend delvinginto any particular subject. Too, there is a much larger universe ofinformation from which the desired information must be extracted. Tryingto get just an overview of a large body of information can beoverwhelming, and attempting to find specific material within the bodyof information can be like searching for a needle in a haystack.

Thus, there is a continuing and growing need for methods and systems forenabling bodies of information to be accessed and reviewed in a usefulmanner, e.g., a manner that allows the scope and content of availableinformation to be quickly ascertained and that enables quick access toinformation of particular interest. In particular, there is a need forsystems and methods of organizing, categorizing and relating the varioussegments of a large body of information to facilitate the access andreview of the body of information. For example, while some previoussystems for enabling observation of a large body of information enableidentification of one or more segments of information that are relatedto a specified segment of information, these systems do notautomatically display such related segments of information. Moreover,the previous systems either require that related segments havepreviously been determined or, at least, that the segments have beencategorized according to subject matter content so that whether twosegments are related can readily be determined. Further, previoussystems have not enabled determination of relatedness between segmentsof information represented by different types of data, e.g., suchsystems cannot determine whether a segment represented by audiovisualdata is related to a segment represented by text data.

There is also a need for systems and methods for enabling observation ofa body of information that are user-friendly, e.g., that can be usedwith little training, that are convenient to use, that enableinformation to be quickly and easily accessed, and that present theinformation in an accessible format via a high quality display medium.It would also be desirable for such systems and methods to be adaptedfor use with bodies of information represented by different types ofdata (i.e., audio data, video data, text data or some combination of thethree). It would further be desirable for such systems and methods to beadapted for use with bodies of information represented by data acquiredfrom a wide variety of media (e.g., print media such as newspapers ormagazines, television and radio broadcasts, online computer informationservices and pre-recorded audiovisual programs, to name a few). Previoussystems and methods for accessing and reviewing a body of informationare deficient in one or more of these respects.

For example, many previous systems are computer-based. Typically, thedisplay device of these systems (e.g., conventional computer displaymonitor) does not provide a high quality display of time-varyingaudiovisual information (such as produced by a television, for example).On the other hand, display devices that do display such information well(e.g., televisions), typically do not provide a high quality display oftext information (such as produced by a computer display monitor). Asystem that can provide a high quality display of both types ofinformation is needed.

Additionally, previous systems for reviewing a body of information arenot as flexible or convenient to use as is desirable. For example, inmany such systems (e.g., computers), the mechanism for controlling theoperation of the system is physically coupled to the display device ofthe system. Therefore, the system can not be operated remotely, thusconstraining the user's freedom of movement while operating the system.Additionally, even in those systems where remote operation is possible(e.g., remotely controlled televisions), the remote control device oftendoes not have a user interface that is as readily accessible as desired(as many consumer electronics users can testify, the keypads of manyremote control devices are an impenetrable array of cryptic controlkeys, often requiring non-intuitive key combinations to effectparticular control instructions) or the remote control device does notcontain a rich set of control features. Moreover, the remote controldevices used with previous systems do not have the capability ofthemselves displaying a part of the body of information.

Further, previous systems often do not enable real-time acquisition andreview of some or all of the body of information. For example, manycomputer-based systems acquire and store data representing a body ofinformation. The stored data can then be accessed to enable display ofsegments of the body of information. However, insofar as previoussystems for observing a body of information allow real-time acquisitionand review of the body of information, these systems generally do notanalyze the data to enable the data to be organized, categorized andrelated so that, for example, segments of the body of information can berelated to other segments for which data is acquired in the future orfor which data has previously been acquired. Moreover, such systems donot enable the real-time display of some or all of a body of informationwhile also displaying related information in response to the real-timedisplay.

Thus, there is a need for improved systems and methods for enablingobservation of a body of information and, in particular, such systemsand methods that address the above-identified inadequacies in previoussystems and methods for enabling observation of a body of information.

SUMMARY OF THE INVENTION

The invention enables a body of information to be displayed byelectronic devices (e.g., a television, a computer display monitor) in amanner that allows the body of information to be reviewed quickly and ina flexible manner. Typically, the body of information will berepresented by a set of audio data, video data, text data or somecombination of the three. In a particular embodiment, the inventionenables generation of an audiovisual display of one or more segments ofinformation, as well as a display (a text display, an audio display, avideo display, or an audiovisual display), for each of the segments, ofone or more related segments of information. In a particular applicationof the invention, referred to herein as a “news browser”, the inventionenables acquisition, and subsequent review, of news stories obtainedover a specified period of time from a specified group of news sources.For example, as a news browser, the invention can be used to review newsstories acquired during one day from several television news programs(e.g., CNN Headline News, NBC Nightly News), as well as from text newssources (e.g., news wire services, traditional print media such asnewspapers and magazines, and online news services such as Clarinet™).

The invention enables some or all of a body of information to be skimmedquickly, enabling a quick overview of the content of the body ofinformation to be obtained. The invention also enables quickidentification of information that pertains to a particular subject. Theinvention further enables quick movement from one segment of a body ofinformation to another, so that observation of particular information ofinterest can be accomplished quickly. In a news browser according to theinvention, for example, each of a set of television news programs can beskimmed to quickly ascertain the subject matter content of the newsstories contained therein. Additionally, a particular category (e.g.,subject matter category) can be specified and news stories havingcontent that fits within the specified subject matter category can beimmediately identified and either displayed or identified as pertinentto the subject matter category and available for display. Further, auser of the news browser can move arbitrarily among news stories withinthe same or different news programs.

The invention also enables automatic identification of information thatis related to information that is being displayed, so that the relatedinformation can be observed, thereby enabling information about aparticular subject to be examined in depth. In particular, the inventionenables such identification of related segments to be made betweensegments of different types (e.g., a segment represented by audiovisualdata can be compared to a segment represented by text data to enable adetermination of whether the segments are related). A portion or arepresentation of the related information can be displayed in responseto (e.g., simultaneous with) the original information display. Forinstance, in a news browser according to the invention, one or more textnews stories (e.g., news stories that are obtained from traditionalprint media or from electronic publications) that are related (i.e.,which cover the same or similar subject matter) to a television newsstory being displayed can be automatically identified and a portion ofthe related text news story or stories displayed so that the story orstories can be reviewed for additional information regarding the subjectmatter of the television news story. Additionally, in a news browseraccording to the invention, one or more other television news storiesthat are related to a television news story being displayed can beautomatically identified and a single representative video framedisplayed for each such news story.

Additionally, the invention enables automatic categorization ofuncategorized segments of the body of information based upon comparisonto other segments of the body of information that have been categorized.In particular, the subject matter category of a segment of informationcan be determined by comparing the segment to one or more previouslycategorized segments and categorizing the segment in accordance with thesubject matter categorization of one or more previously categorizedsegments that are determined to be relevant to the uncategorizedsegment. In a news browser according to the invention, for example, thiscan be used to categorize the news stories of a television news programbased upon the categorization of text news stories that are found to berelevant to the television news stories.

The invention can be implemented in a system that is convenient to use,that presents the body of information in a readily accessible way, andthat presents the information via one or more display devices that aretailored for use with the particular type of data that is used togenerate the display. For example, a system according to the inventioncan include a control device that enables remote, untethered control ofa primary display device of the system. The remote control device canalso be implemented so that some or all of the body of information canalso be displayed on the remote control device. The system can include,for example, a television for display of audiovisual information and acomputer display monitor for display of text information.

Additionally, a control device of a system according to the inventioncan be implemented with a graphical user interface that facilitates userinteraction with the system. For example, such an interface can includea region that provides an indication of a user's past progressionthrough, and present location within, the body of information. In a newsbrowser according to the invention, for example, a program map isdisplayed that facilitates navigation through the news programs that canbe selected for display.

The invention also enables real-time acquisition and review of some orall of the body of information. The invention enables on-the-flyanalysis of data as the data is acquired, so that the data can beorganized, categorized and related to other data. The invention alsoenables the realtime display of some or all of a body of informationwhile also displaying related information in response to the real-timedisplay. For example, in a news browser according to the invention,television news programs can be acquired and displayed as they occur.Related news stories, either from previously acquired television newsprograms or text news sources can be displayed as each television newsstory is displayed in real time.

The invention also enables control of the manner in which theinformation is displayed (e.g., the apparent display rate of the displaycan be controlled, the display can be paused, a summary of a portion ofthe body of information can be displayed). For example, in a newsbrowser according to the invention, the user can cause a summary of oneor more television news stories to be displayed (rather than the entirenews story or stories), the user can speed up (or slow down) the displayof a television news story, and the user can pause and resume thedisplay of a television news story such that the display resumes at anaccelerated rate until the display of the news story “catches up” towhere the display would have been without the pause (a useful featurewhen the television news story is being acquired and displayed in realtime).

In one aspect of the invention, a system enables acquisition and reviewof a body of information that includes a multiplicity of segments thateach represent a defined set of information (frequently, a contiguousrelated set of information) in the body of information. The systemincludes: i) a mechanism for acquiring data representing the body ofinformation; ii) a mechanism for storing the data; iii) a first displaymechanism for generating a display of a first segment of the body ofinformation from data that is part of the stored data; iv) a mechanismfor comparing the data representing a segment of the body of informationto the data representing a different segment of the body of informationto determine whether, according to one or more predetermined criteria,the compared segments are related; and v) a second display mechanism forgenerating a display of a portion of, or a representation of, a secondsegment of the body of information from data that is part of the storeddata. (A method according to the invention, and a computer readablemedium encoded with one or more computer programs according to theinvention, both enable similar capability.) The second display mechanismdisplays a portion or representation of the second segment in responseto the display by the first display mechanism of a first segment towhich the second segment is related The second display mechanism candisplay a portion or representation of the second segment substantiallycoextensive in time with the display of the related first segment by thefirst display mechanism. The system can further include a mechanism foridentifying the subject matter content of a segment of the body ofinformation, so that the mechanism for comparing can determine thesimilarity of the subject matter content of a segment to the subjectmatter content of a different segment (using, for example, relevancefeedback) and use that result to determine the relatedness of thecompared segments. The system can also include a mechanism foridentifying an instruction from a user to begin displaying at least someof the body of information, the first display mechanism beginningdisplay of a segment in response to the user instruction. When a portionor representation of a second segment is being displayed, the system canenable such a second segment to be selected for display by the firstdisplay mechanism. Often, the segments displayed by the first displaymechanism are represented by audiovisual data (and, in particular,audiovisual data that can be used to generate an audiovisual displaythat can vary with time), such as, for example, data produced fromtelevision or radio broadcast signals. The segments displayed by thesecond display mechanism can be represented by audiovisual data (e.g., asingle representative video image, or “keyframe”) or by text data (e.g.,text excerpts), such as, for example, data from computer-readable datafiles acquired over a computer network from an information providingsite that is part of that network. In particular applications for whichuse of the invention is contemplated, the first display mechanism can bean analog display device (such as a television) and the second displaymeans can be a digital display device (such as a computer displaymonitor). The system can advantageously be implemented so that thevarious devices are interconnected to a conventional computer bus thatenables the devices to communicate with each other such that the devicesdo not require wire communication over network communication lines tocommunicate with each other (the devices are “untethered”).

In another aspect of the invention, a system for reviewing a body ofaudiovisual information that can vary with time (e.g., the content fromone or more news broadcasts) includes: i) a mechanism for displaying theaudiovisual information; and ii) a mechanism for controlling operationof the system, the mechanism for controlling being physically separatefrom the mechanism for displaying and including a graphical userinterface for enabling specification of control instructions. Themechanism can advantageously be made portable. Further, the system canadvantageously include a mechanism for 2-way wireless communicationbetween the mechanism for displaying and the mechanism for controlling.The graphical user interface can include one or more of the following:i) a playback control region for enabling specification of controlinstructions that control the manner in which the audiovisualinformation is displayed on the means for displaying; ii) a map regionfor providing a description of the subject matter content of theaudiovisual information and for enabling specification of controlinstructions that enable navigation within the audiovisual information;iii) a related information region for displaying a portion of, or arepresentation of, a segment that is related to a segment beingdisplayed by the mechanism for displaying; and iv) a secondaryinformation display region for displaying a secondary informationsegment that is related to a segment of the audiovisual information thatis being displayed by the mechanism for displaying. In particular, theplayback control region can include one or more of the following: i) aninterface that enables selection of one of a plurality of subject mattercategories, all of the segments of the audiovisual informationcorresponding to a particular subject matter category being displayed inresponse to the selection of that subject matter category; ii) aninterface that enables variation of the apparent display rate at whichthe audiovisual information is displayed; iii) an interface that enablesspecification of the display of a summary of a segment of theaudiovisual information; iv) an interface that enables the display to bepaused, then resumed at an accelerated rate that continues until thedisplay of the audiovisual information coincides with the display thatwould have appeared had the display not been paused; v) an interfacethat enables termination of the current segment display and beginning ofa new segment display; and vi) an interface that enables repetition ofthe current segment display. The map region can further identify asegment of the audiovisual information that is currently being displayedand/or identify each segment of the audiovisual information that haspreviously been displayed.

In still another aspect of the invention, a system enables review of abody of information, the body of information including a first portionthat is represented by audiovisual data that can vary with time and asecond portion that is represented by text data. The system includes afirst display device for displaying the first portion of information anda second display device for displaying the second portion ofinformation. The first display device is particularly adapted forgeneration of a display from time-varying audiovisual data, while thesecond display device is particularly adapted for generation of adisplay from text data. The first display device can be, for example, ananalog display device such as a television. The second display devicecan be, for example, a digital display device such as a computer displaymonitor. The two devices can interact with each other so that relatedinformation can be displayed at the same time on the two devices, in thesame manner as that described above.

In another aspect of the invention, a method categorizes according tosubject matter a segment of a body of information (that includes aplurality of segments), the segment not previously having beencategorized according to subject matter, based upon the subject mattercategory or categories associated with one or more previouslycategorized segments of the body of information. The uncategorizedsegment can have been acquired from a first data source (that supplies,for example, television or radio broadcast signals) and the previouslycategorized segment or segments can have been acquired from a seconddata source (that supplies, for example, computer-readable data files)that is different than the first data source. The method includes thesteps of: i) determining the degree of similarity between the subjectmatter content of the uncategorized segment and the subject mattercontent of each of the previously categorized segments; ii) identifyingone or more of the previously categorized segments as relevant to theuncategorized segment based upon the determined degrees of similarity ofsubject matter content between the uncategorized segment and thepreviously categorized segments; and iii) selecting one or more subjectmatter categories with which to identify the uncategorized segment basedupon the subject matter category or categories used to identify therelevant previously categorized segment or segments. (A computerreadable medium encoded with one or more computer programs according tothe invention enables similar capability.) The step of determining thedegree of similarity can be accomplished using a relevance feedbackmethod. The step of identifying one or more of the previouslycategorized segments as relevant to the uncategorized segment caninclude the steps of: i) identifying a multiplicity of the previouslycategorized segments that are the most similar to the uncategorizedsegment; ii) determining the degree of similarity between each of themultiplicity of previously categorized segments and each other of theplurality of previously categorized segments; iii) for each pair ofpreviously categorized segments of the multiplicity of previouslycategorized segments having greater than a predefined degree ofsimilarity, eliminating one of the pair of previously categorizedsegments from the multiplicity of previously categorized segments,wherein the previously categorized segment or segments remaining afterthe step of eliminating are similar and distinct previously categorizedsegments; and iv) identifying one or more of the similar and distinctpreviously categorized segments as relevant previously categorizedsegments.

In another aspect of the invention, a method determines whether a firstset of information represented by a set of data of a first type (e.g.,text data) is relevant to a second set of information (that is differentthan the first set of information) represented by a set of data of asecond type (e.g., audiovisual data). The method includes the steps of:i) deriving a set of data of the second type from the set of data of thefirst type, the derived set of data of the second type also beingrepresentative of the first set of information; ii) determining thedegree of similarity between the set of data of the second typerepresenting the second set of information and the derived set of dataof the second type representing the first set of information; and iii)determining whether the first set of information is relevant to thesecond set of information based upon the degree of similarity betweenthe set of data of the second type representing the second set ofinformation and the derived set of data of the second type representingthe first set of information. (A computer readable medium encoded withone or more computer programs according to the invention enables similarcapability.) The step of determining the degree of similarity can beaccomplished using a relevance feedback method. Still further inaccordance with this aspect of the invention, a method can determinewhich, if any, of a multiplicity of sets of information represented byan associated set of data of a first type (each of the multiplicity ofsets of information being different from other of the multiplicity ofsets of information) are relevant to the second set of informationrepresented by the set of data of the second type. This method includesthe steps of, in addition to those discussed above: i) determining thedegree of similarity between each set of data of the first typerepresenting one of the multiplicity of sets of information and thederived set of data of the first type representing the second set ofinformation; ii) identifying which, if any, of the sets of data of thefirst type representing one of the multiplicity of sets of informationhave greater than a predefined degree of similarity to the derived setof data of the first type representing the second set of information,the sets of data of the first type so identified being termed similarsets of data of the first type; iii) determining the degree ofsimilarity between each similar set of data of the first type and eachother similar set of data of the first type; iv) for each pair ofsimilar sets of data of the first type having greater than a predefineddegree of similarity, eliminating one of the pair of similar sets ofdata of the first type from the set of similar sets of data of the firsttype, wherein the set or sets of similar data of the first typeremaining after the step of eliminating are similar and distinct sets ofdata of the first type; and v) identifying the set or sets ofinformation corresponding to one or more of the similar and distinctsets of data of the first type as relevant to the second set ofinformation.

In still another aspect of the invention, a method enables theidentification of the boundaries of segments in a body of informationthat is represented by a set of text data and at least one of a set ofaudio data or a set of video data, each segment representing acontiguous related set of information in the body of information. (Acomputer readable medium encoded with one or more computer programsaccording to the invention enables similar capability.) The segmentboundaries are identified by first performing a coarse partitioningmethod to approximately locate the segment boundaries, then performing afine partitioning method to more precisely locate the segmentboundaries. In the coarse partitioning method, time-stamped markers inthe set of text data are identified and used to determine approximatesegment boundaries within the body of information. For each time ofoccurrence of an approximate segment boundary in the text data, a rangeof time is specified that includes the time of occurrence. Subsets ofaudio data or subsets of video data that occur during the specifiedranges of time are extracted from the complete set of audio data or thecomplete set of video data. The fine partitioning method is thenperformed to identify one or more breaks in each of the subsets of audiodata or each of the subsets of video data. The best break that occurs ineach subset of audio data or each subset of video data is selected, andthe time of occurrence of the best break in each subset is designated asa boundary of a segment in the body of information. The finepartitioning can be performed using any appropriate method. For example,when segment boundaries are being determined in video data, scene breakidentification can be used to implement the fine partitioning. Whensegment boundaries are being determined in audio data, the finepartitioning can be implemented by, for example, pause recognition,voice recognition, word recognition or music recognition. Once segmentboundaries have been determined in the audio data or the video data, asynchronization of the audio data and the video data can be used todetermine the boundaries of the segment in the other of the audio dataor video data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a system according to theinvention for acquiring and reviewing a body of information.

FIG. 2A is a diagrammatic representation of a graphical user interfaceaccording to the invention that can be used to enable control of theoperation of a system according to the invention, display informationregarding operation of the system of the invention and displayinformation acquired by the system of the invention.

FIG. 2B is a view of an illustrative graphical user interface inaccordance with the diagrammatic representation of FIG. 2A.

FIG. 3 is a flow chart of a method in accordance with the invention foridentifying the boundaries of segments in a body of information.

FIG. 4 is a flow chart of a method in accordance with the invention fordetermining whether a first set of information represented by data of afirst type is relevant to a second set of information represented bydata of a second type.

FIG. 5 is a flow chart of a method in accordance with the invention forcategorizing according to subject matter an uncategorized segment of abody of information based on the categorization of other previouslycategorized segments of the body of information.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION I. Overview

Generally, the invention enables the acquisition of a body ofinformation and review of the content of the body of information. Inparticular, the invention includes various features that facilitate andenhance review of the body of information. The invention enables thebody of information to be quickly reviewed to obtain an overview of thecontent of the body of information or some portion of the bodyinformation. The invention also allows flexibility in the manner inwhich the body of information is reviewed. For example, the inventionenables a user to move quickly from one segment of a body of informationto another, enabling the user to rapidly begin observing particularinformation of interest. Further, the invention enables a user toquickly locate information within the body of information that pertainsto a particular subject in which the user has an interest. The inventionalso enables a user to, when observing particular information, quicklyfind and review other information that is related to the informationthat the user is observing. Additionally, the invention enables the userto control the manner in which the information is displayed (e.g., theapparent display rate of the display can be controlled, the display canbe paused, a summary of a portion of the body of information can bedisplayed). The invention also provides the user with an indication ofthe user's past progression through, and present location within, thebody of information, such indications aiding the user in selectingfurther segments (described below) of the body of information forreview.

The body of information can be represented by one or more sets of audiodata, one or more sets of video data, one or more sets of text data orsome combination of the three. Herein, “audio data” refers to data usedto generate an audio display, “video data” refers to data used togenerate a video display substantially including images other than textimages, “text data” refers to data used to generate a video (or audio,though typically video) display of text images, and “audiovisual data”refers to data that includes audio and/or video data, and may includetext data. In a particular embodiment, the invention enables theacquisition and review of one or more sets of information represented byaudiovisual data, as well as related sets of information represented bytext data.

For example, in a particular application of the invention, the contentof one or more audiovisual news programs is acquired from a first set ofone or more information sources and news stories (or “articles”) fromtext news sources are acquired from a second set of one or moreinformation sources. The first set of information sources could be, forexample, CNN Headline News or network (e.g., ABC, NBC, CBS) newsprograms. The second set of information sources could be, for example,on-line news services such as Clarinet™ or news wire services such as APor UPI. It is contemplated that this application of the invention can beparticularly useful as a means of enhancing the viewing of conventionaltelevision news programs. For example, in this application, theinvention can enable the user to access the news stories of audiovisualnews programs in a random manner so that the user can move quickly fromone news program to another, or from one news story in a news program toanother news story in the same or another news program. The inventioncan also enable the user to quickly locate news stories pertaining to aparticular subject. Additionally, when the user is observing aparticular news story in an audiovisual news program, the invention canidentify and display a related text news story or stories. The inventioncan also enable the user to control the display of the audiovisual newsprograms by, for example, speeding up the display, causing a summary ofone or more news stories to be displayed, or pausing the display of thenews stories, thereby enabling the user to quickly ascertain the contentof one or more news stories or entire news programs. Additionally, theinvention can indicate to the user which audiovisual news program iscurrently being viewed (and, further, which news story within the newsprogram is being viewed), as well as which news stories and/or newsprograms have previously been viewed.

II. System Configuration

FIG. 1 is a block diagram illustrating a system 100 according to theinvention for acquiring and reviewing a body of information. A user 109interacts with a control device 101 to cause information to be displayedon a primary display device 102. The control device 101 includes anappropriate user interface (e.g., a graphical user interface, asdiscussed in more detail below) that allows the user 109 to specifycontrol instructions for effecting control of the system 100.Communication between the control device 101 and the primary displaydevice 102 is mediated by a system controller 103. The system controller103 causes primary information to be acquired from a primary informationsource 107 via a primary information data acquisition device 105.Herein, “primary information” is any information the display of whichthe user can directly control. The system controller 103 also causessecondary information (which is typically related to the primaryinformation) to be acquired from a secondary information source 108 viaa secondary information data acquisition device 106. Herein, “secondaryinformation” is any information other than primary information that isacquired by a system according to the invention and that can bedisplayed by the system and/or used by the system to manipulate orcategorize (as described in more detail below) the primary information.A data storage device 104 stores the acquired primary and secondaryinformation. The primary information is displayed on the primary displaydevice 102. The secondary information can be displayed (e.g., by thecontrol device 101 or by the primary display device 102 in addition tothe primary information) or not (i.e., the secondary information may beused only for categorizing and/or manipulation of the primaryinformation). Illustratively, the primary information can be videotape(or other audiovisual data representation) of an audiovisual newsprogram or programs and the secondary information can be the text ofnews stories from text news sources.

The control device 101, the primary display device 102, the systemcontroller 103 and the data storage device 104 can be embodied in one ormore devices that can be interconnected to a conventional computer busthat enables the devices to communicate with each other. In particular,the devices 101, 102, 103 and 104 can be integrated into a system inwhich the devices do not require wire communication over networkcommunication lines to communicate with each other (one or more of thedevices 101, 102, 103 and 104 is “untethered” with respect to one ormore of the other devices 101, 102, 103 and 104). Thus, once the primaryand secondary information have been acquired by the system 100, theprimary and secondary information can be accessed and displayed at arelatively fast speed, thus providing quick response to controlinstructions from the user and enabling generation of displays withacceptable fidelity. In contrast, a networked system in which thedevices must communicate with each other over a network via wirecommunication lines—in particular, a system in which the control deviceand display device or devices must communicate over such wirecommunication lines with the data storage device on which theinformation is stored—may not produce acceptable performance. In thenetworked system, the operation of the system is limited by thecommunications bandwidth and latency of the network communicationsmedium. For example, the bandwidth of the network communications mediummay not be adequate to enable transfer of data from the data storagedevice 104 to the primary display device 102 quickly enough to enable adisplay with acceptable fidelity to be generated by the primary displaydevice 102. Or, the response to a control instruction from the controldevice 101 may be undesirably slow because of inadequate speed of thenetwork communications medium.

The primary information data acquisition device 105 and secondaryinformation data acquisition device 106 can be implemented by anyappropriate such devices. Where the primary information source 107 iscomprised of television news broadcasts, for example, the primaryinformation data acquisition device 105 can be a conventional televisiontuner and video capture device that acquires the data representing theprimary information via conventional cable connections, satellite dishor television antenna. Where the secondary information is comprised ofonline text sources (i.e., text sources available over a computernetwork such as the Internet), for example, the secondary informationdata acquisition device 106 can be a conventional modem or othercommunications adapter, as known by those skilled in the art of datacommunications, that enables acquisition of data representing thesecondary information via one or more conventional communication lines,such as telephone lines, ISDN lines or Ethernet connections. (It is alsopossible that the primary information can be acquired from onlinesources, such as via the Internet or other computer network.)

The primary information data acquisition device 105 and the secondaryinformation data acquisition device 106 can communicate with the systemcontroller 103 in any appropriate manner. As described below, the systemcontroller 103 can be implemented as part of a digital computer. Wherethis is the case, the communication between the system controller 103and the devices 105 and 106 is preferably implemented to enable computercontrol of the devices 105 and 106. When the device 105 or 106 is usedto acquire information over a computer network, the device 105 or 106will be a device, such as a computer modem, for which such communicationto the system controller 103 can be implemented using well-known methodsand apparatus For other types of devices, such communication must beimplemented in another manner. For example, when the device 105 is atelevision tuner, communication between the system controller 103 andthe device 105 can be implemented using a VISCA (Video System ControlArchitecture) connection.

As will be apparent from the description below, the processing of thedata representing the primary and secondary information generallyrequires that the data be in digital form. Text data acquired fromonline text sources, for example, is acquired in digital form and so canbe used directly in such processing. Analog television signals, however,must be digitized before being used in digital processing. This can beaccomplished using conventional A/D conversion methods and apparatus.Further, it is desirable to compress the data to increase the amount ofdata (i.e., primary and secondary information) that can be stored on thedata storage device 104. For example, the television data can becompressed according to the MPEG, JPEG or MJPEG video compressionstandards, as known by those skilled in the art of audio and video datacompression. The text data can also be compressed, using conventionaltext file compression programs, such as PKZIP, though, typically, suchcompression provides a relatively small benefit because the amount oftext data is small compared to the amount of audio and video data, andthe amount of data required to represent the categorization information(described below). Finally, it may be desirable or necessary totransform digital data into an analog waveform again (e.g., convertdigital video data into analog video data for display by a television).This can be accomplished using conventional D/A conversion methods andapparatus.

In the embodiment of the invention shown in FIG. 1, the system 100according to the invention makes use of two devices for display andcontrol: a primary display device 102 for displaying the primaryinformation and a control device 101 for controlling the operation ofthe primary display device 102. Preferably, the control device 101 isphysically separate from the primary display device 102 and portable sothat the user has flexibility in selecting a position relative to theprimary display device 102 during use of the system 100. For example,such an embodiment could allow a user to use the invention while sittingin a chair or on a couch, reclining in bed, or sitting at a table ordesk. Additionally, when the secondary information is textual (e.g., thetext of news stories) and the control device 101 is used to display suchsecondary information, the portability of the control device 101attendant such an embodiment increases the likelihood that the text isdisplayed on a device that can be held in close proximity to the user,thereby improving the ability of the user to view the text. Further, asdiscussed in greater detail below, the control device 101 preferably hassophisticated user interface capabilities.

As previously mentioned, a system according to the invention (includingthe system 100) can be implemented so that the primary display device102 displays the primary information while a separate device (e.g., thecontrol device 101) displays the secondary information. Further, as canbe appreciated from the description herein, the invention canadvantageously be used in situations in which the primary information isaudiovisual information (and, in particular, audiovisual informationthat can vary with time, such as the content of a television program)and the secondary information is text information (some or all of whichis, typically, likely to be related to the audiovisual information). Insuch an implementation of the invention, the use of two differentdevices for display allows the optimization of the display devices forthe particular type of information to be displayed. (A system accordingto the invention can, in general, have any number of displays, asnecessary or advantageous.) Thus, where the primary information isaudiovisual information, the primary display device 102 is preferably adevice that enables high quality audio and video images (in particular,time-varying audio and video images) to be produced, such as atelevision. However, while a television is good for displayingaudiovisual information, the television doesn't do as good a job withthe display of text, particularly at typical viewing distances. Acomputer display monitor, on the other hand, does a good job ofdisplaying text. Thus, a computer display monitor can be used to displaythe secondary information. (Herein, a “computer display monitor” candisplay not only video, but also audio.) In particular, a portablecomputer (e.g., a notebook or subnotebook computer) can advantageouslybe used to implement such display. Moreover, the portable computer canalso be used to implement the control device 101, thus allowing thedisplay of the secondary information to be integrated with the userinterface used to specify instructions for controlling operation of thesystem 100. Where a portable computer is used to implement the controldevice 101, communication between the control device 101 and the rest ofthe system 100 is advantageously accomplished using a wireless localarea network (LAN), infrared link, or other wireless communicationssystem, so that the user will have more freedom of movement when usingthe control device 101.

The system controller 103 can be implemented by any conventionalprocessing device or devices that can accomplish the functions of asystem controller as described herein. For example, the systemcontroller 103 can be implemented by a conventional microprocessor chip,as well as peripheral and other computer chips that can be configured toperform the functions of the system controller 103. The data storagedevice 104 can be implemented by any conventional storage devices. Thedata storage device 104 can be implemented, for example, by aconventional computer hard disk (to enable storage of digital data,including analog data—e.g., television or radio signals—that has beendigitized), a conventional videotape (to enable storage of, for example,analog data corresponding to acquired television signals) or aconventional audiotape (to enable storage of, for example, analog datacorresponding to acquired radio signals). In particular, the systemcontroller 103 and data storage device 104 can be implemented, forexample, in a conventional digital computer. The devices with which thesystem controller 103 and data storage device 104 are implemented shouldhave the capability to compress and decompress the audio, video and textdata quickly enough to enable real-time display of that data. The systemcontroller 103 can communicate with the control device 101 and theprimary display device 102 in any appropriate manner, including wire andwireless communications.

In a particular embodiment of the invention, the control device 101 canbe embodied by a portable computer (e.g., a Thinkpad™ computer, made byIBM Corp. of Armonk, N.Y.). The portable computer and associated displayscreen facilitate the presentation of a graphical user interface, aswill be apparent from the description below. Preferably, the portablecomputer has a color display screen. A color display screen furtherfacilitates implementation of a graphical user interface by enablingcolor differentiation to be used to enhance the features provided in thegraphical user interface. The Thinkpad™ can be configured (as known bythose skilled in such art) to act as an X/windows terminal (client) thatcommunicates with an X/windows host (server), using standard X/windowsprotocols (as also known by those skilled in such art), to enablegeneration and display of the graphical user interface. In thisparticular embodiment of the invention, the primary display device 102,as well as the system controller (X/windows host) 103, can be embodied,for example, by an Indigo2 workstation computer made by Silicon GraphicsIncorporated (SGI) of Mountain View, Calif. The portable computer cancommunicate with the SGI Indigo2 computer via a wireless Ethernet link.

Alternatively, both of the primary display device 102 and control device101 could be implemented in a digital computer with the systemcontroller 103 and data storage device 104 (although such animplementation may not have some of the advantages of the embodiments ofthe invention described above). For example, the above-mentioned SGIIndigo2 computer or an IBM-compatible desktop computer could be used toimplement a system of the invention in this manner. In particular,implementation of a system according to the invention in this mannercould advantageously be accomplished on a portable computer such as anotebook computer.

III. User Interface A. Graphical User Interface

1. Overview

FIG. 2A is a diagrammatic representation of a graphical user interface(GUI) 200 according to the invention that can be used to enable controlof the operation of a system according to the invention, displayinformation regarding operation of the system of the invention anddisplay information acquired by the system of the invention. Generally,a GUI according to the invention can be displayed using any suitabledisplay device. Further, when a GUI according to the invention isdisplayed on a display monitor of a digital computer, the GUI can beimplemented by appropriately tailoring conventional computer displaysoftware, as known to those skilled in the art in view of the discussionbelow. For example, the GUI 200 can be displayed on the screen of aportable computer.

The GUI 200 includes four regions: primary information playback controlregion 201, primary information map region 202, related primaryinformation region 203, and related secondary information region 204. Itis to be understood that the regions 201, 202, 203 and 204 could bearranged in a different manner, have different shapes and/or occupy agreater or lesser portion of the GUI 200 than shown in FIG. 2A.Additionally, it is to be understood that a GUI according to theinvention need not include all or any of the regions 201, 202, 203 or204; it is only necessary that the GUI include features that allow thesystem according to the invention to be controlled. Thus, for example, aGUI according to the invention could function adequately without arelated primary information region 203. The GUI also need not, forexample, include a primary information map region 202 or a primaryinformation playback control region 201 having exactly thecharacteristics described below; other interfaces enabling similarfunctionality could also be used. The GUI could also be implemented sothat user interaction with standard GUI mechanisms such as menus anddialog boxes is necessary to cause display of system controls, systemoperation information, and/or acquired information. For example, a GUIaccording to the invention could be implemented such that a display ofthe related secondary information region 204 is produced only uponappropriate interaction with one or more menus and/or dialog boxes.

FIG. 2B is a view of an illustrative GUI 210 in accordance with thediagrammatic representation of FIG. 2A. The GUI 210 is particularlytailored for use with an embodiment of the invention in which theprimary information includes videotape of one or more news programs andthe secondary information includes the text of news stories from textnews sources. Below, the regions 201, 202, 203 and 204 of the genericGUI 200 are described generally, while the corresponding regions 211,212, 213 and 214 of the particular GUI 210 are described in detail.

2. Control of Primary Information Display

The primary information playback control region 201 of the GUI 200 isused to control the manner in which the primary information is displayedon the primary display device 102. The region 201 can be used, forexample, to provide a mechanism to enable the user to begin, stop orpause display of the primary information, as well as rewind or fastforward the display. The region 201 can also be used, for example, tocontrol the particular primary information that is displayed, as well asthe apparent display rate at which the primary information is displayed.

As seen in FIG. 2B, the primary information playback control region 211of the GUI 210 includes topic “buttons” 215, control “buttons” 216 and aspeed control 217. It is to be understood that the functionality of thetopic buttons 215, control buttons 216 and speed control 217, describedbelow, could be accomplished in a manner other than that shown in FIG.2B and described below.

The topic buttons 215 enable the user to select a subject mattercategory so that, for example, all news stories in the recorded newsprograms that pertain to the selected subject matter category aredisplayed one after the other by the primary display device 102.Alternatively, selection of a topic button 215 could cause a list ofnews stories pertaining to that subject matter category to appear, fromwhich list the user could select one or more news stories for viewing.(The categorization of the primary information by subject mattercategory is discussed in more detail below.) The GUI 210 includes sixtopic buttons 215 to enable selection of news stories related tointernational news (“World”), national news (“National”), regional news(“Local”), business news (“Business”), sports news (“Sports”), and humaninterest news (“Living”); however, a GUI according to the invention caninclude any number of topic buttons and each button can correspond toany desired subject matter category designation.

The control buttons 216 enable the user to control which news story isdisplayed, as well as the manner in which a news story is displayed.Moving from left to right in FIG. 2B, the control buttons 216respectively cause the display to activate a dialog box that enables theuser to perform a keyword search of the text of news stories acquired bythe system of the invention, return to the beginning of the currentlydisplayed story to begin displaying the story again, stop the display,start the display, and skip ahead to the next story in a predeterminedsequence of stories. A GUI according to the invention can include othercontrol buttons that enable performance of other functions instead of,or in addition to, the functions enabled by the control buttons 216,such as fast forwarding the display, rewinding the display, pausing thedisplay (a particular method according to the invention is describedbelow), and displaying a summarized version of the primary information(a particular method according to the invention is described in moredetail below).

The speed control 217 can be used to increase or decrease the apparentdisplay rate with which the primary information is displayed. The speedcontrol display 217 shows a number that represents the amount by which anormal display rate is multiplied to produce the current apparentdisplay rate, and includes a graphical slider bar that can be used toadjust the apparent display rate. The manner in which the apparentdisplay rate can be changed is described in more detail below.

3. Map of Primary Information Display

The primary information map region 202 of the GUI 200 provides the userwith a description of the content of the primary information that isavailable for display, as well as information that facilitatesnavigation through the primary information, and can also be used toallow the user to select particular primary information for display. Thedescription of the primary information can include, for example, anillustration or other description of the subdivision of the primaryinformation into smaller portions (e.g., segments) of information. Suchillustration or description can convey the number of portions, thelength (i.e., time duration) of each portion and the subject matter ofeach portion. The region 202 can also be used to show the user thelocation within the primary information of the portion of the primaryinformation that is currently being viewed, as well as which (if any)portions of the primary information have previously been viewed.Additionally, the region 202 can be used to enable the user to movefreely among portions of the primary information by, for example, usinga conventional mouse to point and click on a portion of the primaryinformation that is illustrated in the region 202.

As seen in FIG. 2B, the primary information map region 212 of the GUI210 includes several subdivided rows, each row representing a particularnews program (e.g, CNN Headline News, NBC Nightly News, etc.). Each rowis a map that illustrates to some level of detail the content of thecorresponding news program. Each of the subdivisions of a row representbreaks during the news program, such as breaks between news stories. Theregion between each subdivision represents a news story (a region couldalso represent, for example, an advertisement). The duration of eachnews story is depicted graphically by the length of the regioncorresponding to that news story. Each region in a row can be displayedin a particular color, each color representing a particularpredetermined subject matter category (i.e., topic), so that the colorof each region denotes the subject matter category of the news storycorresponding to that region.

The map region 212 can be further enhanced in any of a variety of ways.For example, the news program (row) that is currently being viewed canbe marked, such as by, for example, shading the row of the currentlyviewed news program a particular color or causing a particular type ofsymbol to appear adjacent to the row of the currently viewed newsprogram. Additionally, news stories that have already been viewed can bemarked in an appropriate manner, such as by, for example, causing theregions of the viewed news stories to be cross-hatched or to be shaded aparticular color. The current viewing location can also be shown: inFIG. 2B, this is shown by a vertical line.

4. Related Primary Information

The related primary information region 203 of the GUI 200 displays“thumbnails” which identify segments of the primary information that arerelated to the primary information that is currently being displayed.Though the region 203 includes four thumbnails 203 a, 203 b, 203 c, 203d, generally, the region 203 can be used to display any number ofthumbnails. Further, the thumbnails can take any form, such as a displayof a portion of the segment or a display of a representation of thesegment. For example, the thumbnails 203 a, 203 b, 203 c, 203 d can besingle video images that represent the video data of the segment beingidentified (“keyframes”). (As seen in FIG. 2B, the related primaryinformation region 213 of the GUI 210 includes three single video imagesthat each represent a news story from a news program.) Alternatively,the thumbnails 203 a, 203 b, 203 c, 203 d could be a text summary orother text identifier of the segment being identified. Or, thethumbnails 203 a, 203 b, 203 c, 203 d could be pictorial representationsthat identify the corresponding segment. Other possibilities exist, asknown to those skilled in the art.

To enable display of thumbnails, primary information segments that arerelated to the primary information segment that is being displayed mustbe determined. A threshold of relatedness (the expression of thethreshold depending upon the method used to determine relatedness) ispreferably specified so that only segments that are sufficiently relatedto the displayed segment are displayed in the related primaryinformation region 203, even if that means that less than the allottednumber of segments (including no segments) are displayed. Ifappropriate, redundant segments can be eliminated from the primaryinformation segments to be displayed in the related primary informationregion 203, using techniques similar to those described below foreliminating redundant segments from a set of segments identified assimilar to a designated segment (e.g., eliminating redundant secondaryinformation segments that are similar to a displayed primary informationsegment).

Identification of the relatedness of primary information segments can beaccomplished by determining the degree of similarity between the primaryinformation segment being displayed and each other primary informationsegment. The degree of similarity can be determined using anyappropriate method, such as, for example, relevance feedback. The use ofrelevance feedback to determine the similarity between two segments isdiscussed in more detail below with respect to the determination of therelatedness of primary and secondary information segments (see, inparticular, section IV.B.2. below). The use of relevance feedbacknecessitates that sets of text data that represent the primaryinformation segments be created (by, for example, using a conventionalspeech recognition method to create a transcript of the spoken portionof the audio data set) if such sets of text data do not already exist(e.g., a closed-caption transcript).

When the thumbnails 203 a, 203 b, 203 c, 203 d are keyframes, eachkeyframe should be representative of the video content of the segmentbeing identified. Each keyframe can be, for example, a video frameselected from the video data representing the segment. The keyframe canbe selected from the video data in any appropriate manner.

For example, the keyframe can be a video frame that occurs at aspecified location within the video data of the segment In a particularembodiment of the invention in which the primary information comprisestelevision news stories, a video frame that occurs one tenth of the waythrough the video data representing the news story is selected. Onetenth was chosen because it was determined empirically that video framesof particular relevance to the content of a television news story tendto occur at about that point in the television news story.

Alternatively, the keyframe can be selected based upon an analysis ofthe content of the video data. One method of accomplishing this isdescribed in detail in the commonly owned, co-pending U.S. patentapplication entitled “A Method of Compressing a Plurality of VideoImages for Efficiently Storing, Displaying and Searching the Pluralityof Video Images,” by Subutai Ahmad, U.S. Ser. No. 08/528,891, filed onSep. 15, 1995, the disclosure of which is incorporated by referenceherein. In that method, the content of each video frame is representedby a vector. The vector can comprise, for example, the discrete cosinetransform (DCT) coefficients for the video frame, as known to thoseskilled in the art of video image analysis. (The DCT coefficientsindicate, for example, how much objects in a video frame have movedsince the previous video frame.) From the vectors for all of the videoframes of the video data of the segment an average vector is determined.The keyframe is selected as the video frame that is represented by avector that is closest to the average vector for the video data. Thismethod of selecting a keyframe can be advantageous as compared to thearbitrary selection of a video frame that occurs at a specified locationwithin the video data, since it is likely to result in the selection ofa video frame that is more representative of the video content of thesegment.

Rather than selecting a single video frame from the video data to be thekeyframe, multiple keyframes can be identified from the video data andthe keyframes “tiled,” i.e., presented together adjacent to each other.Or, the video data can be analyzed and a composite video framesynthesized from the video data. Any technique for synthesizing a videoframe or frames can be used.

The keyframe may also be a video frame or frames that are not selectedfrom the video data. For example, a representative video image (e.g.,one or more video frames) can be selected from a library of videoimages. For instance, a news story about baseball could be representedby a keyframe showing a batter swinging at a pitch. Such selection canbe done manually, i.e., at some point, a person reviews or is made awareof the content of the segment and, based upon that knowledge, associatesa video image from the library with the segment. Alternatively, suchselection can be accomplished automatically (meaning, here, withouthuman intervention, except to establish the criteria for the selectionprocess) by analyzing the audiovisual data of the segment (e.g., with anappropriately programmed digital computer) to ascertain the content ofthe segment and, based upon that analysis, associating a video imagefrom the library with the segment. The content of the segment could bedetermined, for example, using a categorization method as described inmore detail below. The segment to be categorized could either becompared to previously categorized segments that can be displayed by thesystem of the invention, or to a library of “control segments”, each ofwhich contain words germane to a particular subject.

The GUI 200 can be implemented, using conventional interface methods, sothat a user of a system of the invention can select (e.g., by pointingand clicking with a mouse) one of the thumbnails 203 a, 203 b, 203 c,203 d to cause the corresponding primary information segment to bedisplayed. (The map in the primary information map region 202 isadjusted accordingly.)

5. Related Secondary Information

The related secondary information region 204 of the GUI 200 provides theuser information from a secondary information source or sources, thesecondary information being related to the primary information currentlybeing displayed. Though the region 204 includes two secondaryinformation displays 204 a, 204 b, generally, the region 204 can includeany number of secondary information displays. Further, as with thethumbnails 203 a, 203 b, 203 c, 203 d of the related primary informationregion 203, the secondary information displays 204 a, 204 b can take anyform. For example, the secondary information displays 204 a, 204 b couldbe single video images, moving video images or sets of text. (As shownin FIG. 2B, the related secondary information region 214 of the GUI 210includes three sets of text that each are a story from a text newssource.) Other possibilities exist for the secondary informationdisplays 204 a, 204 b, as known to those skilled in the art. As thesegment of primary information being displayed changes, the secondaryinformation displays 204 a, 204 b typically change as well. As indicatedabove, segments of secondary information that are related to the primaryinformation that is being displayed can be identified in a mannerdiscussed in more detail below. The system according to the inventioncan also be implemented so that the user can cause various parts of thesecondary information displays 204 a, 204 b to be displayed, e.g., theuser can be enabled to scroll up and down through a set of text or moveback and forth through a video clip, using conventional GUI tools suchas mouse pointing and clicking.

B. Other User Interface Techniques

User interface techniques other than GUI can be used with the invention.For example, rather than using GUI “buttons” (as illustrated in theprimary information playback control region 211 of the GUI 210 of FIG.2B), the manner in which the primary information is displayed could becontrolled using a rotating knob device. Rotation of the knob in onedirection could cause the display of the primary information to moveforward (play); rotation of the knob in the other direction could causethe display of the primary information to move backward (rewind).Further, the knob could be constructed so that as the knob is rotatedthe user feels detents at certain points in the rotation. Each detentcould correspond to a particular apparent display rate of the display.For example, when the knob is positioned in a home position, the displayis stopped. When the knob is rotated clockwise, the display movesforward, the first detent in the clockwise direction causing the displayto occur at a normal display rate, the second detent specifying a targetapparent display rate of, for example, 1.5 times the normal displayrate, the third detent specifying a target apparent display rate of, forexample, 2.0 times the normal display rate, and so on. Similarly, whenthe knob is rotated counterclockwise, the display moves backward (i.e.,in a chronological direction opposite that in which the display normallyprogresses). The first detent corresponds to normal display rate, thesecond detent specifies a target display rate of, for example, 1.5 timesthe normal display rate, and so on. The maximum rotation of the knob ineither direction could be limited, the maximum rotation corresponding toa maximum target apparent display rate. The knob could be positioned atany position in between, thus allowing the target apparent display rateto be varied continuously between the maximum forward and backwarddisplay rates. The knob could also include a centrally locatedpushbutton to, for example, enable skipping from the display of onesegment of the primary information to a next segment of the primaryinformation. The knob could be constructed so that the position of theknob (or activation of the pushbutton) is transmitted to the remainderof the system using wireless communications, thus providing the userwith relatively large freedom of movement during use of the system.

IV. Processing of Obtained Information

A. Information Acquisition

1. In General

Returning to FIG. 1, the system controller 103 causes data to beacquired from the primary information source 107 and the secondaryinformation source 108, as described above. The data is acquired usingmethods and apparatus that are appropriate to the type of data beingacquired. For example, the system controller 103 can acquire datarepresenting television broadcasts using conventional equipment forreceiving (e.g., a television set and antenna) and recording (e.g., aconventional videocassette recorder) television signals. Or, the systemcontroller 103 can acquire data representing radio broadcasts usingconventional equipment for receiving (e.g., a radio and antenna) andrecording (e.g., a conventional audiotape recorder) radio signals. Or,the system controller 103 can acquire computer-readable data files (thatcan include text data, audio data, video data or some combination of twoor more of those types of data), using conventional communicationshardware and techniques, over a computer network (e.g., a public networksuch as the Internet or a proprietary network such as America Online™,CompuServe™ or Prodigy™) from an information providing site that is partof that network. In one particular embodiment of the invention, thesystem controller 103 acquires primary information including thetelevision signals representing the content of designated televisionnews broadcasts, and secondary information including computer-readabledata files that represent the content of designated news stories fromtext news sources.

The data can be acquired according to a pre-established schedule (thatcan be stored, for example, by the data storage device 104). Data can beacquired at any desired frequency and the scheduled acquisition timesspecified in any desired manner (e.g., hourly, daily at a specifiedtime, weekly on a specified day at a specified time, or after theoccurrence of a specified event). The schedule can be used, for example,to program a videocassette recorder to record particular televisionprograms at particular times. Likewise, the schedule can be used, forexample, to appropriately program a computer to retrieve desired datafiles from particular network sites (e.g., by specifying an appropriatenetwork address, such as a URL) of a computer network at specifiedtimes. In the latter case, if the device with which the systemcontroller 103 is implemented is not operating (e.g., the computer isnot turned on) at a time when a scheduled acquisition of data is to takeplace, the system controller 103 can be implemented so that all suchdata is immediately retrieved upon beginning operation of the device(e.g., turning the computer on). Further, connection over the network tothe site or sites from which data is to be obtained can be accomplishedby, for example, inserting a communications daemon into a startup filethat is executed at the beginning of operation of the operating systemof a computer used to implement the system controller 103. For example,if the computer uses a Windows operating system, the daemon can initiatea WinSock TCP/IP connection to enable connection to be made to thenetwork site.

The acquired data must be stored. As indicated above, analog data (suchas television or radio signals) can be stored on an appropriate medium,such as videotape or audiotape. Additionally, some or all of the dataacquired by a system according to the invention is, if not already inthat form, converted to digital data. The digital data can be stored ona conventional hard disk having adequate capacity, as described above.To minimize the amount of data storage capacity required, the digitaldata can be compressed using conventional techniques and equipment.Illustratively, a half hour television news program requiresapproximately 250 MB of hard disk storage capacity when the video isrecorded using Adobe Premiere with Radius Studio compression at 15 fpsand “high” quality capture at 240×180 resolution, and the audio isrecorded at approximately 22 kHz.

Appropriate rules can be established to handle situations in which thedata storage device 104 (whether single or multiple devices) hasinsufficient data storage capacity to store new data. For example, theoldest data can be deleted, as necessary, to make room for new data. Forexample, in the particular embodiment of the invention in which theprimary information is the content of designated television newsprograms and the secondary information is the content of designated textnews stories, as new television news programs are recorded, the oldeststored programs can be deleted as necessary to make space to store thenew programs, and text stories that are older than a specified length oftime (e.g., several days) are automatically deleted.

The GUI 200 (FIG. 2A) can also include a mechanism for enabling the userto specify the particular information desired, i.e., specify particularinformation providers (e.g., news networks, such as CNN, NBC, ABC orCBS, or information services, such as Clarinet™) and data acquisitionschedules for both the primary information source 107 and the secondaryinformation source 108. This could be implemented, for example, using aset of nested menus, as known by those skilled in the art.

2. Recording/Playback Mediation

A system according to the invention may be instructed to acquire newinformation at the same time that the system is instructed to displayother information. However, limitations of the devices or configurationof the system of the invention can impede or prevent such simultaneousacquisition and display. For example, the operating speed of a hard diskused to store the data describing the acquired information can limit thecapacity of the system for such simultaneous operation: for typicalamounts of audiovisual data, current conventional hard disks may notoperate at a speed that is adequate to enable the simultaneous storingof data to, and accessing of stored data from, the hard disk.

Thus, in one embodiment of the invention, when data acquisition isscheduled to begin at a time when the system of the invention is beingused for information display, a conventional graphical user interfacemechanism (e.g., a dialog box) is used to alert the user of the systemto the conflict and offer a choice between continuing with the display(thus delaying or eliminating the data acquisition) or ending thedisplay and allowing the data acquisition to occur.

In another embodiment of the invention, the user can be alerted oL animpending data acquisition at some predetermined time before the dataacquisition is scheduled to begin. Similar to the choice describedabove, the user can be presented with a choice to continue with thedisplay at that time or allow the data acquisition to occur. The systemof the invention can default to one or the other modes of operation(i.e., data acquisition or display) if the user does not make aselection.

Or, the hard disk operating speed limitation described above can bealleviated or overcome by using multiple hard disks so that if dataacquisition begins at a time when data is being accessed for use ingenerating a display, the newly acquired data is stored to a hard diskthat does not contain any previously stored data (or that, based uponevaluation of one or more predetermined rules, does not contain datathat is expected to be accessed during the time that the new data isbeing acquired), thus ensuring that data access and data storage willnot occur simultaneously for a single hard disk. Alternatively, the harddisk operating speed limitation can be addressed by using only someportion of the available data to generate the information display, thusfreeing more time for use in storing data to the hard disk. However,this latter approach may decrease the fidelity of the displayunacceptably.

In a similar approach to the two hard disk approach described above, thedata being acquired can be stored on a data storage device of one type,while the data to be used for generating a display is accessed from adata storage device of another type. For example, incoming televisionsignals could be stored on a videocassette tape by a VCR, while digitaldata from previous television transmissions is retrieved from a harddisk for use in generating a television display of the previouslyacquired data. The data recorded by the VCR could be digitized at alater time and stored on the hard disk for subsequent use (which use mayalso occur at a time at which incoming television signals are beingacquired by the VCR).

B. Information Structuring

Typically, the data representing the primary and secondary informationare not provided from the primary and secondary information sources in aform that enables the various aspects of the invention described hereinto be realized. Thus, it is necessary or desirable to “structure” thedata (i.e., to organize and categorize the data, and relate particulardata to other data) in useful ways. Below are described several aspectsof such data structuring that can be implemented as part of theinvention.

1. Partitioning

The primary and secondary information can be, and typically are, divided(“partitioned”) into smaller related sets of information of particularutility for the invention is the identification within the primary andsecondary information of contiguous related sets of information thattypically concern a single theme or subject and that can be delineatedin some manner from adjacent information. Herein, each such contiguousrelated set of information can be referred to as a “segment” of theprimary or secondary information. (Note that, in the descriptionbelow—see section IV.C.1.—of skimming an audiovisual display, “segment”is used in a different way; there, “segment” represents a contiguousportion of a set of audio data that occurs during a specified durationof time.) Segments within the primary information are “primaryinformation segments” while segments within the secondary informationare “secondary information segments.” For example, if the primaryinformation includes the content of several news programs, the primaryinformation can be divided into particular news programs and each newsprogram can further be broken down into particular news stories withinthe news program, each news story being denoted as a segment. Similarly,if the secondary information includes content from several text sources,the secondary information can be divided into particular text sourcesand each text source can be further divided into separate text stories,each text story being denoted as a segment. Note that a “segment” maysometimes, strictly speaking, not be contiguous in time (though it iscontiguous in content). For example, a news story that is interrupted bya commercial break, then continues after the commercial break, may bedefined as a single segment, particularly if the body of information ismodified so that commercial breaks—and other extraneous portions of thebody of information—are eliminated (an approach that, generally, ispreferred, though such portions could also be treated as segments).

Partitioning the primary and secondary information into segments isuseful for a variety of reasons For example, each segment of the primaryinformation can be identified within the data storage device whichstores the data representing the primary information, in a manner knownby those skilled in the art (e.g., by maintaining a table of segmentidentifiers and associated locations of the beginning of the identifiedsegment), thus enabling the primary information segments to be accessedrandomly so that the user can change the displayed segment freely amongthe primary information segments. Such identification of primaryinformation segments also enables the creation of the map region 202 ofthe GUI 200 (FIG. 2). Further, each segment of the primary informationcan be correlated, as described in more detail below, with segments ofthe secondary information, thereby enabling one or more secondaryinformation segments that are sufficiently related to a primaryinformation segment to be displayed at the same time that the primaryinformation segment is displayed. As also described in more detailbelow, the correlation of primary information segments with secondaryinformation segments can also be used to categorize the primaryinformation segments according to subject matter, thus enabling the userto sort or to cause display of segments of the primary information thatpertain to a particular subject matter category (see the discussion ofthe topic buttons 215 in the playback control region 211 of the GUI 210shown in FIG. 2A).

Generally, partitioning of a set of data requires some analysis of thedata to identify “breaks” within the data, i.e., differences betweenadjacent data that are of sufficient magnitude to indicate a significantchange in the content of the information represented by the data. Abreak may signify a demarcation of one segment from another, but neednot necessarily do so: a break may also signify, for example, a changein the video image within a segment or a change of speakers within asegment. Methods for enabling identification of breaks that constitutesegment demarcation are discussed in more detail below.

Partitioning of text data is often straightforward. For example, bodiesof information that are collections of segments (e.g., stories) fromtext sources that are represented as computer-readable data typicallyinclude markers that identify the breaks between segments. Similarly,text transcripts of bodies of information represented as a set ofaudiovisual information also frequently include markers that identifybreaks between segments of the information. For example, closed captiontext data that can accompany the audio and video data of a set ofaudiovisual data often includes characters that indicate breaks in thetext data (most news broadcasts, for example, include closed captiontext data containing markers that designate story and paragraphboundaries, the beginning and end of advertisements, and changes inspeaker) and, in particular, characters that explicitly designate breaksbetween segments (e.g., markers that identify story boundaries).Partitioning of such text data, then, requires only the identificationof the location (e.g., if the text transcript of a set of audiovisualdata is time-stamped, the time of occurrence) of the markers within thetext data.

Where such markers are not present, the text data can be partitionedbased upon analysis of the content of the text data. In a set ofaudiovisual data, breaks between segments can be determined, forexample, based upon identification of the occurrence of a particularword, sequence of words, or pattern of words (particularly words thattypically indicate a transition), and identification of changes inspeaker. As one illustration, in a news program, phrases of the form,“Jane Doe, WXYZ news, reporting live from Anytown, USA,” can indicate abreak between segments.

Partitioning of audio and video data typically requires some non-trivialanalysis of the data. The partitioning of audio and video data inaccordance with the invention can be accomplished in any suitablemanner. Some examples of methods that can be used to accomplishpartitioning of audio or video data are described below. (These methodsare applicable to digital data; thus, if the primary information isinitially analog, it must be digitized before partitioning.) Typically,the audio and video data are synchronized as a result of having beenrecorded together. Thus, partitioning of either the audio or the videodata will result in a corresponding partitioning of the other of theaudio and video data. However, if the audio and video data are notsynchronized, then such synchronization must be accomplished, inaddition to partitioning one of the audio or video data, so that theother of the audio and video data can be partitioned in like manner.

Partitioning of audio data can be accomplished in any of a number ofways. For example, the audio data can be partitioned using a known voicerecognition method. A voice recognition method that could be used withthe invention is described in “A Gaussian Mixture Modeling Approach toText-Independent Speaker Identification,” by Douglas Reynolds, PhDthesis, Dept. of Electrical Engineering, Georgia Institute ofTechnology, 1992, the disclosure of which is incorporated by referenceherein. Voice recognition methods can be tailored to, for example,identify a break in the audio data when a particular voice speaks, whena particular sequence of voices speak, or when a more complicatedoccurrence of voices is identified (e.g., the occurrence of two voiceswithin a specified time of each other, or the occurrence of a voicefollowed by a silence of specified duration). Illustratively, when theinvention is implemented as a news browser, a break between news storiescould be identified when a particular newscaster's voice is followed orpreceded by a silence of specified duration.

Or, the audio data can be partitioned using a known word recognitionmethod. For example, a conventional speech recognition method (a largevariety of which are known to those skilled in that art) can be used toenable identification of words. The identified words can then beanalyzed in the same manner as that described above for analysis of textdata, e.g., transition words or speaker changes can be used to indicatebreaks. Illustratively, when the invention is implemented as a newsbrowser, a break between news stories could be identified when one of aset of particular word patterns occurs (e.g., “we go now to”, “updatefrom”, “more on that”).

Audio data can also be partitioned using music recognition, i.e., abreak is identified when specified music occurs. A method forpartitioning audio data in this way is described in detail in thecommonly owned, co-pending U.S. patent application entitled “System andMethod for Selective Recording of Information,” by Michelle Covell andMeg Withgott, U.S. Ser. No. 08/399,482, filed on Mar. 7, 1995, thedisclosure of which is incorporated by reference herein. Partitioning ofaudio data using music recognition can be particularly useful whentransitions between segments of the body of information are sometimesmade using standard musical phrases. Illustratively, when the inventionis implemented as a news browser, music recognition can be used topartition certain news programs (e.g., The MacNeil/Lehrer news hour)which use one or more standard musical phrases to transition betweennews stories.

Another method for partitioning audio data is pause recognition. Pauserecognition is based on the assumption that a pause occurs at the timeof a significant change in the content of the primary information. Formany types of information, such as news programs, this is a workableassumption. A break is identified each time a pause occurs. A pause canbe defined as any period of silence having greater than a specifiedmagnitude.

Video data can be partitioned, for example, by searching for scenebreaks, a method similar to the pause recognition method forpartitioning audio data discussed immediately above. One method ofaccomplishing this is described in detail in the above-mentioned U.S.patent application entitled “A Method of Compressing a Plurality ofVideo Images for Efficiently Storing, Displaying and Searching thePlurality of Video Images,” by Subutai Ahmad. In that method, thecontent of each video frame is represented by a vector, as describedabove. The vector for each video frame is compared to the vector of theimmediately previous video frame and the immediately subsequent videoframe, i.e., vectors of adjacent video frames are compared. In oneapproach, a break is identified each time the difference between thevectors of adjacent video frames is greater than a predeterminedthreshold. In another approach, a predetermined number of partitions isspecified and the video frames are partitioned to produce that number ofpartitions (the partitioning can be accomplished by considering eachvideo frame to be initially partitioned from all other video frames andrecursively eliminating the partition between partitioned video frameshaving the least difference, or considering none of the video frames tobe partitioned and recursively establishing partitions betweenunpartitioned video frames having the greatest difference).

Other approaches to scene break identification could be used, as knownby those skilled in the art of processing video images. Some otherapproaches to scene break identification are discussed in “AutomaticParsing of News Video,” by HongJiang Zhang, Gong Yihong, Stephen W.Smoliar, and Tan Ching Yong, IEEE Conference on Multimedia Computing andSystems, Boston, May 1994, the disclosure of which is incorporated byreference herein. For example, scene breaks could be identified basedupon the magnitude of the overall changes in color of the pixels ofadjacent video frames (a color change having a magnitude above aspecified threshold is identified as a scene break). Or, scene breakscould be identified based upon the magnitude of the compression ratiofor a particular set of adjacent video frames (a relatively small amountof compression indicates a relatively large change between video framesand, likely, a change in scenes, i.e., a scene break).

The above-described methods for partitioning audio or video datadirectly may not, by themselves, enable identification of segment breaksto be accomplished easily or at all. For example, without augmentation,pause recognition or scene break identification typically are notimplemented in a manner that enables distinguishing between segmentbreaks and other breaks. Voice recognition may not, alone, be a reliableindicator of segment breaks, since switches in speaker often occur forreasons unrelated to a segment break. Word recognition, too, may beerratic in determining segment breaks; it also requires obtaining a texttranscript of the audio. Music recognition works well only with alimited number of information sources, i.e., information sources thatuse well-defined musical transitions.

It may be possible to include markers (similar to those discussed abovewith respect to closed caption text data) in either audio or video datathat directly identify segment or other breaks within the audio or videodata. The invention contemplates use of such markers to segment audioand/or video data.

If a set of audiovisual data also includes text data (e.g., a closedcaption transcript of the spoken audio), it is possible to partition theaudiovisual data by partitioning the text data, then using thepartitioned text data to partition the audio data and video data in acorresponding manner. Even if the audiovisual data does not initiallyinclude text data, the text data can be produced using a speechrecognition method. The text data can be partitioned using anyappropriate method, as described above.

Typically, the text data, audio data and video data are eachtime-stamped. Theoretically, then, once segment breaks are determined inthe text data, the time-stamps of the beginning and end of each segmentwithin the text data could be used directly to identify segment breakswithin the audio data and/or video data. However, in practice, the textdata is typically not exactly synchronized with the audio data and videodata (e.g., the text data of a particular segment may begin or endseveral seconds after the corresponding audio or video data), makingsuch a straightforward approach infeasible. Nevertheless, thetime-stamps of the segment breaks in the text data can be used to enablesynchronization of those segment breaks with the corresponding segmentbreaks in the audio and video data. Such synchronization can beaccomplished using any appropriate technique. Some possible approachesare described below.

One way to partition the audio and video data based upon the partitionof the text data is to use a synchronization of the complete set ofaudio data with the complete set of text data, and a synchronization ofthe complete set of audio data with the complete set of video data toidentify the partitions in the audio and video data. The lattersynchronization typically exists as a consequence of the manner in whichthe audio and video data is obtained. However, synchronization betweenthe text data and the audio data frequently does not already exist, and,if it does not, obtaining such synchronization can be computationallyexpensive. Further, it is not necessary to synchronize all of the textdata with the audio and video data, but, rather, only the locations ofthe segment breaks.

A simpler approach is to determine the segment breaks in the audio andvideo data from the segment breaks in the text data based upon a rule orrules that exploit one or more characteristics of the body ofinformation. Such a rule might be based on an observation that segmentbreaks in the audio and/or video data of a set of audiovisual data beara relatively fixed relationship to the corresponding segment breaks inthe corresponding text data. For example, it was observed that the videodata of a news story from an audiovisual news program frequently beginsabout 5 to 10 seconds before the closed caption text data of the newsstory. Thus, in one embodiment of news browser implementation of theinvention, the beginning of the video data of a news story is assumed tobe 4 seconds prior to the closed-caption text data. This enables most ofthe relevant video data to be captured, while reducing the possibilityof capturing extraneous video. This approach was found to be accuratewithin 2 seconds for CNN Headline News and the news programs of the NBC,ABC and CBS television broadcasting networks.

In some cases, the approach may still not produce as good a result asdesired, i.e., the segmentation of the audio and video data is not ascrisp as desired, either deleting part of the beginning or end of theaudio or video segment, or including extraneous audio or video as partof the segment. Thus, according to another particular embodiment of theinvention, partitioning of audiovisual data that includes text data inwhich segments breaks are explicitly designated by markers within thetext data can be accomplished in two steps: a first, coarse partitioningfollowed by a second, fine partitioning. FIG. 3 is a flow chart of amethod 300, in accordance with this aspect of the invention, foridentifying the boundaries of segments in a body of information. In thecoarse partitioning step 301 of the method 300, the time-stampsassociated with the segment breaks in the text data can be used toapproximate the location of the corresponding segment breaks in theaudio and video data, as described above. In step 302, a window of data(e.g., audio or video data in the context of the current discussion)that includes the approximate segment boundary is specified. This can beaccomplished, for example, by specifying a time range that includes thetime associated with the segment break in the text data (e.g., the timeof occurrence of the segment break in the text data plus or minusseveral seconds) and identifying audio and/or video data that fallswithin that time range from the time-stamps associated with the audioand/or video data. The fine partitioning step 303 can then be used toidentify breaks within the audio and/or video data. The finepartitioning can be accomplished using any appropriate method, such asone of the above-discussed methods (i.e., scene break identification,pause recognition, voice recognition, word recognition, or musicrecognition) to identify breaks in audio and video data. The finepartitioning can be performed on the entire set of audio data or videodata, or only on the audio or video data that occurs within the timerange. In the step 304, the data within the time range can then beexamined to identify the location of a break or breaks within the timerange. If more than one break is identified, the “best” break, measuredaccording to the criteria of the partitioning method used, can beidentified as the segment break, or the break occurring closest in timeto the approximate segment break can be identified as the segment break.

Once the segment breaks in the audio or video data are identified,segment breaks in the other of the audio or video data can be determinedusing a synchronization of the audio and video data, as discussed above.Pointers to the segment breaks in the text data, audio data and/or videodata can be maintained to indicate the beginning and end of eachsegment, thus enabling random access to segments within a body ofinformation (e.g., news stories within a news program), as discussed inmore detail above. The identified segments can also be used to enableother features of the invention, as described in more detail below.

2. Correlation

As mentioned above, the related secondary information region 204 of theGUI 200 is used to provide the user, from a secondary information sourceor sources, information that is related to the primary informationcurrently being displayed. Thus, it is necessary to determine which ofthe segments of the secondary information are sufficiently related tothe primary information segment displayed on the primary display device102 to be displayed in the related secondary information region 204.This can be accomplished by determining the degree of similarity betweeneach segment of the primary information (e.g., news story from anaudiovisual news program) and each segment of the secondary information(e.g., text story from a text news source), and displaying in therelated secondary information region 204 of the GUI 200 certainsecondary information segments that are most similar to the primaryinformation segment that is being displayed by the primary displaydevice 102.

An important aspect of the invention is the capability to determinerelatedness of segments of information represented by two differenttypes of data. In particular, the invention can enable the determinationof relatedness between segments of information represented byaudiovisual data (such as is frequently the case for the primaryinformation that can be displayed by the invention) and segmentsrepresented by text data (such as is generally the case for thesecondary information as described particularly herein). This aspect ofthe invention enables the display of the related secondary informationregion 204 to be generated. It can also enable categorization ofuncategorized segments, as described further below.

FIG. 4 is a flow chart of a method 400, in accordance with this aspectof the invention, for determining whether a first set of informationrepresented by a first set of data of a first type (e.g., audiovisualdata) is relevant to a second set of information represented by a secondset of data of a second type (e.g., text data). In step 401, a set ofdata of the second type is derived from the first set of data of thefirst type. In a typical application of the method 400, step 401 causesa set of text data to be produced from a set of audiovisual data. Theset of text data can be produced in any appropriate manner. For example,“production” of the set of text data may be as simple as extracting apre-existing text transcript (e.g., a closed caption transcript) fromthe set of audiovisual data. Or, the set of text data can be producedfrom the set of audio data using a conventional speech recognitionmethod. In step 402, the derived set of data (of the second type) iscompared to the second set of data of the second type to determine thedegree of similarity between the derived set of data and the second setof data. One way of making this determination is described in moredetail below. In step 403, a determination is made as to whether thefirst set of data is relevant to the second set of data, based on thecomparison of step 402. Typically, a threshold level of similarity (theexpression of which depends upon the method used to determinesimilarity) is specified so that only sets of information that aresufficiently related to each other are identified as related. (Thismeans, when the method 400 is used to generate the related secondaryinformation region 204, that less than the allotted number of secondaryinformation segments—or even no secondary information segments—may bedisplayed.)

The degree of similarity can be determined using any appropriate method,such as, for example, relevance feedback. In relevance feedback, a textrepresentation of each segment to be compared (e.g., each audiovisualnews story or text story) is represented as a vector, each component ofthe vector corresponding to a word, the value of each component beingthe number of occurrences of the word in the segment. (Two words areconsidered identical—i.e., are amalgamated for purposes of ascribing amagnitude to each component of the vector representing the textualcontent of a segment—if the words have the same stem; for example,“play”, “played” and “player” are all considered to be the same word forpurposes of forming the segment vector.) For each pair of segments, thenormalized dot product of the vectors corresponding to the segments iscalculated, yielding a number between 0 and 1. The degree of similaritybetween two segments is represented by the magnitude of the normalizeddot product, 1 representing two segments with identical words and 0representing two segments having no matching words. The use of relevancefeedback to determine the similarity between two text segments iswell-known, and is described in more detail in, for example, thetextbook entitled Introduction to Modern Information Retrieval, byGerard Salton, McGraw-Hill, New York, 1983, the pertinent disclosure ofwhich is incorporated by reference herein. Relevance feedback is alsodescribed in detail in “Improving Retrieval Performance by RelevanceFeedback,” Salton, G., Journal of the American Society for InformationScience, vol. 41, no. 4, pp. 288-297, June 1990 as well as “The Effectof Adding Relevance Information in a Relevance Feedback Environment,”Buckley, C. et. al., Proceedings of 17th International Conference onResearch and Development in Information Retrieval, DIGIR 94,Springer-verlag (Germany), 1994, pp. 292-300, the disclosures of whichare incorporated by reference herein.

The related secondary information region 204 of the GUI 200 can displaya predetermined number of relevant secondary information segments.Generally, it is desirable to display the secondary information segmentsthat are most similar to the primary information segment that is beingdisplayed. While this can be accomplished straightforwardly bydisplaying those secondary information segments having the highestdetermined degree of similarity, such an approach may not be desirablein some situations. For example, the secondary information source mayinclude segments that are identical or nearly identical (e.g., newsstories are often repeated in a variety of text news sources with littleor no change), so that display of the secondary information segmentshaving the highest determined degree of similarity can result inundesirable redundancy.

This problem can be overcome by further determining the degree ofsimilarity between each of a predetermined number of the secondaryinformation segments having the highest determined degree of similarity(in one embodiment of the news browser implementation of the invention,the 10 most similar text stories are compared), and displaying only oneof each pair of secondary information segments having a degree ofsimilarity above a specified threshold, i.e., redundant secondaryinformation segments are eliminated. Again, this can be more problematicthan first appears. For example, a particular segment may have greaterthan the threshold degree of similarity when compared to each of secondand third segments, but the second and third segments may have less thanthe threshold degree of similarity when compared to each other. From thethree segments, it would be desirable to show both the second and thirdsegments. However, if the first segment is compared to the secondsegment or the third segment, and the second or third segment discarded,before comparison of the first segment to the other of the second orthird segment (which will also result in discarding of one of thecompared segments), then only one of the three segments will be shown.Such a situation could be handled by, for example, calculating thesimilarity between all pairs of the predetermined number of secondaryinformation segments, and performing comparisons that reveal thesituation described above before discarding any of the secondaryinformation segments.

3. Categorizing

An important aspect of the invention is the capability to categorizeuncategorized segments of information based upon the categorization ofpreviously categorized segments of information. In particular, if thesegments of the secondary information have been categorized according tosubject matter, then the degree of similarity between the subject mattercontent of segments of the primary information (e.g., news stories inaudiovisual news programs) and segments of the secondary information(e.g., news stories from text news sources) can also be used tocategorize the primary information according to subject matter. This canbe useful to enable determination of which primary information segmentsfall within a particular subject matter category that corresponds to oneof the topic buttons 215 (FIG. 2) that a user can select to cause allprimary information segments that pertain to the selected subject mattercategory to be displayed one after the other by the primary displaydevice 102 (FIG. 1). Though this aspect of the invention has particularutility in categorizing primary information segments based upon thecategorization of pre-existing secondary information segments, it cangenerally enable any categorized segments to be used to categorizeuncategorized segments.

FIG. 5 is a flow chart of a method 500, in accordance with this aspectof the invention, for categorizing according to subject matter anuncategorized segment of a body of information based on the subjectmatter categorization of other previously categorized segments of thebody of information. For example, each story from the Clarinet™ newsservice is categorized according to the subject matter of the story byassociating one or more predefined subject matter categories (e.g.,sports, travel, computers, business, international news) with the story.This subject matter categorization can be used to categorize newsstories from audiovisual news programs based on the similarity betweeneach audiovisual news story and text stories from the Clarinet™ newsservice. Below, such categorization of audiovisual news stories isdescribed as an example of how categorizing segments of primaryinformation can be accomplished in accordance with the invention.

The subject matter category or categories associated with each Clarinet™text story are acquired as part of the acquisition of the text storiesthemselves and can, for example, be stored in a relational database in amemory that is part of the system controller 103 (FIG. 1). It may bedesirable to associate only one subject matter category with each textstory. For example, the most salient subject matter category can beidentified in any appropriate manner and used as the sole subject mattercategory associated with the story. This may be done., for example, toincrease the likelihood that the subject matter category eventuallyassociated with each news story accurately describes the subject mattercontent of that news story.

In step 501 of the method 500, a determination is made as to the degreeof similarity between the subject matter content of an uncategorizedsegment and that of previously categorized segments. The degree ofsimilarity can be determined using any appropriate method, such as, forexample, relevance feedback. When relevance feedback is used, it isnecessary to obtain a textual representation of audiovisual data, ifappropriate (i.e., if one or both of the segments is represented asaudiovisual data) and not already existent.

In step 502, previously categorized segments that are relevant to theuncategorized segment are identified. Relevant segments can beidentified based upon the degree of similarity in the same manner asthat described above with respect to correlation of segments, e.g.,segments having greater than a threshold level of similarity can bedesignated as relevant. Step 501 can also include elimination ofredundant segments (in the same manner as described above) from amongthose that have the required degree of similarity to the uncategorizedsegment.

In step 503, the uncategorized segment is categorized based upon thesubject matter categories associated with the relevant previouslycategorized segments. One or more subject matter categories can beassociated with the uncategorized segment. Generally, the subject mattercategory or categories can be selected from the subject mattercategories associated with the relevant previously categorized segmentsusing any desired method. For example, the subject matter category orcategories of the most similar previously categorized segment could beselected as the subject matter category or categories of theuncategorized segment. Or, the most frequently occurring subject mattercategory or categories associated with a predefined number of the mostsimilar previously categorized segments (or previously categorizedsegments having greater than a threshold degree of similarity) could beselected as the subject matter category of the uncategorized segment. Inthe latter case, it may be particularly desirable, as described above,to determine the similarity between the relevant previously categorizedsegments, so that only one of a set of previously categorized segmentsthat are substantially identical to each other influences thecategorization of the uncategorized segment.

C. Information Presentation

Above, the acquisition of information and the structuring of acquiredinformation has been described. The information must, of course, also bedisplayed to a user. The information display has been describedgenerally above with respect to FIGS. 2A and 2B. However, a systemaccording to the invention can also include one or more of a variety ofadditional features that enhance the information display.

1. Skimming

As indicated above with respect to FIGS. 2A and 2B, the apparent displayrate with which the primary information is displayed by the primarydisplay device 102 can be varied by the user. Variation in the apparentdisplay rate of an audiovisual display can be implemented byappropriately programming a digital computer to accomplish the functionsof a method for varying the apparent display rate. Generally, any methodfor varying the apparent display rate can be used with the invention. Asdescribed elsewhere herein, the primary information will often berepresented by coextensive sets of data of several types (audio, videoand, possible text). The particular method used to vary the apparentdisplay rate of the primary information will typically depend upon thetype of the set of data (e.g., audio, video, text) that is directlymodified to produce appropriately modified data for use in generating adisplay of the primary information at the new apparent display rate. Themethod also preferably synchronizes the sets of data that are notdirectly modified with the set of data that is.

For example, the audio data can be modified to cause the apparentdisplay rate of the audio display to be varied (either slowed down orspeeded up) from a normal display rate and the video data synchronizedwith the modified audio data (resulting in a variation of the apparentvideo display rate that corresponds to the variation in the apparentaudio display rate). Several methods of accomplishing such variation inthe apparent display rate of an audiovisual display are described indetail in the commonly owned, co-pending U.S. patent applicationentitled “Variable Rate Video Playback with Synchronized Audio,” by NealA. Bhadkamkar, Subutai Ahmad and Michelle Covell, attorney docket numberI0359-991160, filed on the same day as the present application, thedisclosure of which is incorporated by reference herein. At least someof the methods described therein have the advantage that the apparentdisplay rate of the audio can be varied while maintaining proper pitch(i.e., the voices don't sound stupefied when the display is slowed downor like chipmunks when the display is speeded up) and, therefore,intelligibility. A brief description of a general method describedtherein is given immediately below, followed by a brief description ofone particular method for modifying the audio data.

Generally, in the methods described in the above-mentioned patentapplication, a correspondence between an original audio data set and anoriginal video data set is first established. For example, the number ofaudio samples that have the same duration as a frame of video data canbe determined and that number of audio samples defined to be an audiosegment. (Note that, as mentioned above, as used here in the descriptionof skimming, “segment” refers to a contiguous portion of a set of audiodata that occurs during a specified duration of time; elsewhere herein,“segment” refers to a contiguous related set of information within theprimary or secondary information that typically concerns a single themeor subject and that can be delineated in some manner from adjacentinformation.) The audio segments can be defined, for example, so thateach audio segment corresponds to a single particular video frame. Atarget display rate (which can be faster or slower than a normal displayrate at which an audiovisual display system generates an audiovisualdisplay from the unmodified, original sets of audio and video data) isalso determined. The target display rate can be a single value whichremains unchanged throughout the display or a sequence of values suchthat the target display rate changes during the display. The originalaudio data set is manipulated, based upon the target display rate and anevaluation of the original audio data set, to produce a modified audiodata set. As described below, the modified audio data set is produced sothat, generally, when the modified audio data set is used to generate anaudio display, the audio display appears to be speeded up or slowed downby an amount that is approximately equal to the target display rate. Thecorrespondence between the modified audio data set and the originalaudio data set, and the correspondence between the original audio dataset and the original video data set, are used to create a correspondencebetween the modified audio data set and the original video data set,which, in turn, is used to delete video data from, or add video data to,as appropriate, the original video data set to create a modified videodata set. Once the modified audio and video data sets have been created,an audiovisual display can be generated from those modified data sets byan audiovisual display system, or the modified audio and video data setscan be stored on a conventional data storage device for use ingenerating a display at a later time. The audio and video data of themodified audio and video data sets are processed at the same rate asbefore (i.e., when the original audio and video data sets were used togenerate a display at the normal display rate) by the audiovisualdisplay system. However, since the modified audio and video data sets(in the usual case) have a different amount (either more or less) ofdata than the original audio and video data sets, the apparent displayrate of the audiovisual display generated from the modified audio andvideo data sets is different than the normal display rate. Further,since the modified video data set is created based upon the content ofthe modified audio data set and a correspondence between the modifiedaudio data set and the original video data set, the modified video dataset is synchronized (at least approximately and, possibly, exactly) withthe modified audio data set and produces a display of the same orapproximately the same duration.

The audio data can be modified in any suitable manner; one way isdescribed following. An audio data set is divided into non-overlappingsegments of equal length. Generally, the beginning and end of eachsegment are overlapped with the end and beginning, respectively, ofadjacent segments. (Note that the overlap can be negative, such that thelength of the adjacent segments is extended. The audio data ofcorresponding overlapped portions of adjacent segments are blended andreplaced by the blended audio data. The possible lengths of each overlapare constrained in accordance with a target overlap that corresponds tothe specified target display rate. However, within this constraint, thelength of each particular overlap is chosen so that the pitch pulses ofthe overlapped portions closely resemble each other. Consequently, theblending of the audio data of the overlapped portions does not greatlydistort the sound corresponding to the overlapped portions of audiodata. Thus, the invention enables the audio data set to be condensed orexpanded a desired amount (i.e., the display of an audio data set can bespeeded up or slowed down as desired), while minimizing the amount ofdistortion associated with the modification of the audio data set (i.e.,the audio display sounds “normal”).

Since the actual amount of overlap of segments can vary from the targetoverlap that corresponds to the specified target display rate, theactual apparent display rate can vary from the target display rate. Overrelatively long periods of time (e.g., greater than approximately 0.5seconds), the actual apparent display rate typically closelyapproximates the target display rate. Over shorter time periods (e.g.,approximately 30 milliseconds), the actual apparent display rate canvary more substantially from the target display rate. However, theseshort term fluctuations are not perceptible to an observer. Thus, thismethod produces an actual apparent display rate that to an observerappears to faithfully track the target display rate over the entirerange of the display.

Preferably, the computation required to produce a particular amount ofvariation in the apparent display rate is done at the time that thedetermination of a target display rate mandates such variation. This hasthe advantage of reducing the amount of data storage capacity requiredby a system of the invention. This also enables any magnitude ofapparent display rate to be specified over a continuous range of alloweddisplay rates, rather than restricting the magnitude of the apparentdisplay rate to one of a set of discrete magnitudes within an allowedrange, as would be necessary if all of the computations for eachmagnitude of apparent display rate were pre-computed. Additionally, thisenables the apparent display rate of the display to be varied in realtime.

2. Summarization

A system according to the invention can include another informationpresentation feature that enables the display of a primary segment orsegments to be summarized. Summarization enables an observer to quicklyget an overview of the content of a particular segment or segments ofinformation. Summarization can be implemented by appropriatelyprogramming a digital computer to accomplish the functions of asummarization method. Generally, summarization can be accomplished usingany appropriate method. As with skimming, discussed above, theparticular method used will typically depend upon the type of the set ofdata (e.g., audio, video, text) that is directly modified to produceappropriately modified data for use in generating a summary display ofthe primary information. The method also preferably synchronizes thesets of data that are not modified directly with the set of data thatis.

For example, text data that is part of, or derived from, audiovisualdata that represents a primary segment can be summarized, and thecorresponding audio and video data summarized based upon the textsummary. One method of accomplishing such summarization is described indetail in the commonly owned, co-pending U.S. patent applicationentitled “Indirect Manipulation Of Data Using Temporally Related Data,With Particular Application To Manipulation Of Audio Or AudiovisualData,” by Emanuel E. Farber and Subutai Ahmad, attorney docket numberI0359-991110, filed on the same day as the present application, thedisclosure of which is incorporated by reference herein. A briefdescription of that method is given immediately below.

The text data of a set of audiovisual data represents a transcription ofthe spoken portion of the audio data and is temporally related to eachof the audio and video data. The text data can be obtained in anyappropriate manner, e.g., the text data can be pre-existing text datasuch as closed-caption data or subtitles, or the text data can beobtained by using any of a number of known speech recognition methods toanalyze the audio data to produce the text data.

The text data is summarized using an appropriate summarization method.Generally, any text summarization method can be used; a particularexample of a text summarization method that can be used with theinvention is described in U.S. Pat. No. 5,384,703, issued to Withgott etal. on Jan. 24, 1995.

The unsummarized text data is aligned with the unsummarized audio data.If the text data has been obtained from the audio data using a speechrecognition method, then the alignment of the unsummarized text datawith the unsummarized audio data typically exists as a byproduct of thespeech recognition method. Otherwise, alignment is accomplished in threesteps. First, the unsummarized text data is evaluated to generate acorresponding linguistic transcription network (e.g., a networkdescribing the set of possible phonetic transcriptions). Second, afeature analysis is performed on the audio samples comprising theunsummarized audio data set to create a set of audio feature data.Third, the linguistic transcription network is compared to the set ofaudio feature data (using Hidden Markov Models to describe thelinguistic units of the linguistic transcription network in terms ofaudio features) to determine the linguistic transcription (from all ofthe possible linguistic transcriptions allowed by the linguistictranscription network) which best fits the set of audio feature data. Asa result of this comparison, the audio features of the best fitlinguistic transcription are correlated with audio features in the setof audio feature data. The audio features of the best fit linguistictranscription can also be correlated with the linguistic units of thelinguistic transcription network. The linguistic units of the linguistictranscription network can, in turn, be correlated with the unsummarizedtext data. As a consequence of these correlations, an alignment of theunsummarized text data with the unsummarized audio data can be obtained.Using the previously determined text summary and the alignment betweenthe text data and audio data, an audio summary can be produced.

A video summary can be produced from the audio summary using analignment between the unsummarized audio data and the unsummarized videodata. Such alignment can be pre-existing (because the audio data andvideo data were recorded together, the alignment being inherent becauseof the like time stamps associated with each of the audio and videodata) or can be calculated easily (the time stamp for an audio sample orvideo frame can be calculated by multiplying the time duration of eachsample or frame by the sequence number of the sample or frame within theaudio data or video data).

Another method that can be used to summarize the display of a set ofaudiovisual information includes identifying and eliminating “soundbites” (defined below) in the audio portion of the primary information.The sound bites can be identified based upon analysis of a set of textdata that corresponds to the spoken portion of the set of audio data.The text data can be obtained in any appropriate manner. For example,the text data may be closed caption data that is provided with the audioand video data representing the primary information. Or, the text datacan be obtained from the set of audio data using conventional speechrecognition techniques. Once the text data is obtained, the text datacan be “pre-processed” using known methods to classify the words in thetext data according to their characteristics, e.g., part of speech.

Herein, a “sound bite” is a related set of contiguous audio informationthat conforms to one or more predetermined criteria that are intended toidentify short spoken phrases that are not spoken by a previouslyidentified primary speaker and that represent information of littleinterest and/or are redundant For example, in a news browser accordingto the invention, where the primary information includes the content ofaudiovisual news programs (e.g., television news programs), thepredetermined criteria can be established so that spoken portions of theaudio information that are likely not to have been spoken by a newsanchorperson or a news reporter are identified as sound bites. Suchcriteria might include, for example, rules that tend to identify aspoken portion of the audio as a sound bite if the spoken portionincludes slang words or the use of first person pronouns (e.g., I orwe), both of which tend not to be present in the speech of ananchorperson or reporter. As can be appreciated, elimination of suchaudio portions will typically not significantly adversely affect thepresentation of the essential content of a set of audio information, butwill enable the set of audio information to be presented more quickly.(It should be noted that the summarization method of Withgott et al. wasalso found to be incidentally effective at eliminating sound bites.)

Once the audio data has been modified by eliminating the audio datacorresponding to the sound bites, the set of modified audio data must bealigned (synchronized) with the video data (if present) to enable thevideo data to be modified to produce a speeded-up video display. Asdescribed above with respect to the summarization method of Farber andAhmad, the audio/video alignment can either be pre-existing orcalculated easily.

As can be appreciated, a summarization method such as one of thosedescribed above could be used in combination with a method forincreasing the apparent display rate as described above (see sectionIV.C.1. above on Skimming) to even further condense the display of a setof primary information. For example, the set or sets of datarepresenting the primary information could be modified to increase theapparent display rate, then the modified set or sets of data could besummarized to produce a speeded-up summary of the set of primaryinformation. Or, conversely, the set or sets of data representing theprimary information could be summarized, then the summarized set or setsof data modified to increase the apparent display rate, thus producing aspeeded-up summary of the set of primary information.

As can be appreciated, the methods described above for manipulatingaudiovisual data to produce a summarized display of the audiovisual datacan also be used, with appropriate modification (e.g., instead ofproducing a summary of the text data, the text data could be manipulatedin some other desired fashion), to manipulate the audiovisual data forsome other purpose, such as rearranging, editing, selectively accessingor searching the audiovisual data.

3. Display Pause with Elastic Playback

A system according to the invention can include yet another informationpresentation feature that enables the display of an image to be paused,then, at the end of the pause, resumed at an accelerated rate (i.e., arate that is faster than a normal display rate) until a time at whichthe content of the display corresponds to the content that would havebeen displayed had the image been displayed at the normal display ratewithout the pause, at which time display of the image at the normaldisplay rate resumes. In other words, after a pause, the image displayis speeded up so that the display “catches up” to where it would havebeen without the pause, then slowed back down to the normal displayrate. The implementation of this feature is described in detail in thecommonly owned, co-pending U.S. patent application entitled “DisplayPause with Elastic Playback,” by Subutai Ahmad, Neal A. Bhadkamkar,Steve B. Cousins, Paul A. Freiberger and Brygg A. Ullmer, attorneydocket number I0359-991150, filed on the same day as the presentapplication, the disclosure of which is incorporated by referenceherein. A brief description of the implementation is given immediatelybelow.

The image to be displayed is represented by an ordered set of displaydata. This display data is acquired from a data source at a first rate.The display data is transferred to a display device at the first rate asthe display data is acquired. An image is generated from the displaydata transferred to the display device and displayed on the displaydevice. At some point, the user instructs the system to pause thedisplay. The system identifies the pause instruction from the user and,in response, stops the transfer of display data to the display deviceand begins storing the acquired display data at the first rate. At somelater time, the user instructs the system to resume the display. Thesystem identifies the resume instruction from the user and, in response,begins transferring stored display data to the display device at asecond, effective rate that is greater than the first rate. An image isgenerated from the stored display data transferred to the display deviceand displayed on the display device. While the stored display data isbeing transferred to the display device, the newly acquired datacontinues to be stored. The storage of display data finally stops whenthere is no more stored display data to be transferred to the displaydevice, the amount of stored display data having gradually been reducedby transferral of the stored display data to the display device at thesecond, effective rate that is greater than the first rate at which thedisplay data is stored. Once the storage of display data stops, thedisplay data is again transferred to the display device at the firstrate as the display data is acquired.

This feature of the invention enables a great deal of flexibility inobserving a real-time display of audiovisual information. For example,the invention enables an observer to pause and resume the display asdesired so that, if the observer wants to temporarily stop watching togo to the bathroom or to take a phone call, the observer can pause thedisplay, then, after resuming the display upon return, watch theaudiovisual information at an accelerated display rate until the displayof the program catches up to where it would have been without the pause.Thus, the user can attend to other matters while the audiovisualinformation is being viewed, without sacrificing viewing any of thecontent of the audiovisual information or enduring the inconvenience ofspending additional time to finish watching the audiovisual program.This feature of the invention can also be tailored to enable a user whohas begun viewing the audiovisual information at a time later thandesired, to observe the audiovisual information at an accelerated rateuntil the display catches up to the point at which the display have beenif the audiovisual information had been viewed at a normal display ratebeginning at the desired start time.

Various embodiments of the invention have been described. Thedescriptions are intended to be illustrative, not limitative. Thus, itwill be apparent to one skilled in the art that certain modificationsmay be made to the invention as described without departing from thescope of the claims set out below.

We claim:
 1. A system for acquiring and reviewing a body of information,wherein the body of information includes a plurality of segments, eachsegment representing a defined set of information in the body ofinformation, the system comprising: means for acquiring datarepresenting the body of information; means for storing the acquireddata; first display means for generating a display of a first segment ofthe body of information from data that is part of the stored data; meansfor comparing data representing a segment of the body of information todata representing a different segment of the body of information todetermine whether, according to one or more predetermined criteria, thecompared segments are related; and second display means for generating adisplay of a portion of, or a representation of, a second segment of thebody of information from data that is part of the stored data, whereinthe second display means displays the portion or representation of thesecond segment in response to the display by the first display means ofa first segment to which the second segment is related.
 2. A system asin claim 1, wherein the second display means displays the portion orrepresentation of the second segment substantially coextensive in timewith the display of the related first segment by the first displaymeans.
 3. A system as in claim 1, wherein: at least a portion of thebody of information is represented by audiovisual data; the firstsegment is represented by audiovisual data; the first display meansdisplays an audiovisual display of the first segment; and the secondsegment is represented by audiovisual data.
 4. A system as in claim 3,further comprising means for selecting a segment for which a portion orrepresentation is displayed by the second display means, whereinselection of such segment causes the first display means to display anaudiovisual display of the selected segment.
 5. A system as in claim 1,wherein: at least a portion of the body of information is represented byaudiovisual data; the first display means displays an audiovisualdisplay of the first segment; and the second display means displays atext display of a portion or representation of the second segment.
 6. Asystem as in claim 1, wherein: the first display means is an analogdisplay device; and the second display means is a digital displaydevice.
 7. A system as in claim 1, wherein: the first display means is atelevision; and the second display means is a computer display monitor.8. A system as in claim 1, further comprising means for identifying thesubject matter content of a segment of the body of information, whereinthe means for comparing further comprises means for determining thesimilarity of the subject matter content of a segment to the subjectmatter content of a different segment, the predetermined criteriaincluding a predefined degree of similarity with respect to which therelatedness of the compared segments is determined.
 9. A system as inclaim 8, wherein the means for determining the similarity of the subjectmatter of segments further comprises means for performing a relevancefeedback method.
 10. A system as in claim 1, wherein the means foracquiring data further comprises means for acquiring televisionbroadcast signals.
 11. A system as in claim 1, wherein the means foracquiring data further comprises means for acquiring radio broadcastsignals.
 12. A system as in claim 1, wherein the means for acquiringdata further comprises means for acquiring computer-readable data filesover a computer network from an information providing site that is partof that network.
 13. A system as in claim 1, wherein the means foracquiring data further comprises: means for acquiring televisionbroadcast signals; and means for acquiring computer-readable data filesover a computer network from an information providing site that is partof that network.
 14. A system as in claim 13, wherein: the first segmentis represented by data produced from the television broadcast signals;and the second segment is represented by data from the computer-readabledata files.
 15. A system as in claim 1, further comprising means foridentifying an instruction from a user to begin displaying at least someof the body of information, wherein the first display means beginsdisplaying a segment in response to the user instruction.
 16. A systemas in claim 1, wherein the first and second display means are physicallyseparate.
 17. A system as in claim 1, wherein the means for storing theacquired data, the first display means and the second display means areinterconnected to a conventional computer bus that enables the devicesto communicate with each other such that the devices do not require wirecommunication over network communication lines to communicate with eachother.
 18. A system as in claim 1, wherein at least some of the acquireddata is digital data, the means for acquiring data further comprisingmeans for acquiring digital data.
 19. A system as in claim 1, wherein atleast some of the acquired data is analog data, the means for acquiringdata further comprising means for acquiring analog data.
 20. A methodfor acquiring and reviewing a body of information, wherein the body ofinformation includes a plurality of segments, each segment representinga defined set of information in the body of information, the methodcomprising the steps of: acquiring data representing the body ofinformation; storing the acquired data; generating a display of a firstsegment of the body of information from data that is part of the storeddata; comparing data representing a segment of the body of informationto data representing a different segment of the body of information todetermine whether, according to one or more predetermined criteria, thecompared segments are related; and generating a display of a portion of,or a representation of, a second segment of the body of information fromdata that is part of the stored data, wherein the display of the portionor representation of the second segment is generated in response to thedisplay of a first segment to which the second segment is related.
 21. Amethod as in claim 20, further comprising the step of causing thedisplay of the portion or representation of the second segment to occursubstantially coextensive in time with the display of the related firstsegment.
 22. A method as in claim 20, wherein: the step of acquiringdata representing the body of information further comprises the step ofacquiring audiovisual data representing at least a portion of the bodyof information, wherein the first and second segments are represented byaudiovisual data; and the step of generating a display of a firstsegment of the body of information further comprises the step ofgenerating an audiovisual display of the first segment.
 23. A method asin claim 22, further comprising the step of identifying the selection ofa second segment for which a portion or representation is beingdisplayed, wherein selection of such second segment causes anaudiovisual display of the selected second segment to be produced.
 24. Amethod as in claim 20, wherein: the step of acquiring data representingthe body of information further comprises the step of acquiringaudiovisual data representing at least a portion of the body ofinformation; the step of generating a display of a first segment of thebody of information further comprises the step of generating anaudiovisual display of the first segment; and the step of generating adisplay of a portion of, or a representation of, a second segment of thebody of information further comprises the step of generating a textdisplay of the portion or representation of the second segment.
 25. Amethod as in claim 20, wherein: the step of generating a display of afirst segment of the body of information further comprises the step ofgenerating a display of the first segment on an analog display device;and the step of generating a display of a portion of, or arepresentation of, a second segment of the body of information furthercomprises the step of generating a display of the portion orrepresentation of the second segment on a digital display device.
 26. Amethod as in claim 20, wherein: the step of generating a display of thefirst segment on an analog display device further comprises the step ofgenerating a display of the first segment on a television; and the stepof generating a display of the portion or representation of the secondsegment on a digital display device further comprises the step ofgenerating a display of the portion or representation of the secondsegment on a computer display monitor.
 27. A method as in claim 20,further comprising the step of identifying the subject matter content ofa segment of the body of information, wherein the step of comparingfurther comprises the step of determining the similarity of the subjectmatter content of a segment to the subject matter content of a differentsegment, the predetermined criteria including a predefined degree ofsimilarity with respect to which the relatedness of the comparedsegments is determined.
 28. A method as in claim 27, wherein the step ofdetermining the similarity of the subject matter of segments furthercomprises the step of performing a relevance feedback method.
 29. Amethod as in claim 20, wherein the step of acquiring data furthercomprises the step of acquiring television broadcast signals.
 30. Amethod as in claim 20, wherein the step of acquiring data furthercomprises the step of acquiring radio broadcast signals.
 31. A method asin claim 20, wherein the step of acquiring data further comprises thestep of acquiring computer-readable data files over a computer networkfrom an information providing site that is part of that network.
 32. Amethod as in claim 20, wherein the step of acquiring data furthercomprises the steps of: acquiring television broadcast signals; andacquiring computer-readable data files over a computer network from aninformation providing site that is part of that network.
 33. A method asin claim 32, wherein: the first segment is represented by data producedfrom the television broadcast signals; and the second segment isrepresented by data from the computer-readable data files.
 34. A methodas in claim 20, further comprising the step of identifying aninstruction from a user to begin displaying at least some of the body ofinformation, wherein the display of a first segment is begun in responseto the user instruction.
 35. A method as in claim 20, wherein the firstand second segments are displayed on physically separate displaydevices.
 36. A method as in claim 20, wherein the steps of storing theacquired data, generating a display of a first segment of the body ofinformation, and generating a display of a portion of, or arepresentation of, a second segment of the body of information areperformed by devices interconnected to a conventional computer bus thatenables the devices to communicate with each other such that the devicesdo not require wire communication over network communication lines tocommunicate with each other.
 37. A method as in claim 20, wherein atleast some of the acquired data is digital data, the step of acquiringdata further comprising the step of acquiring digital data.
 38. A methodas in claim 20, wherein at least some of the acquired data is analogdata, the step of acquiring data further comprising the step ofacquiring analog data.
 39. A method for categorizing according tosubject matter an uncategorized segment of a body of information thatincludes a plurality of segments, each segment representing a definedset of information in the body of information, one or more segments ofthe body of information having previously been categorized byidentifying each of the one or more segments with one or more subjectmatter categories, the method comprising the steps of: determining thedegree of similarity between the subject matter content of theuncategorized segment and the subject matter content of each of thepreviously categorized segments; identifying one or more of thepreviously categorized segments as relevant to the uncategorized segmentbased upon the determined degrees of similarity of subject mattercontent between the uncategorized segment and the previously categorizedsegments; and selecting one or more subject matter categories with whichto identify the uncategorized segment based upon the subject mattercategories used to identify the relevant previously categorizedsegments.
 40. A method as in claim 39, wherein the step of determiningthe degree of similarity is accomplished using a relevance feedbackmethod.
 41. A method as in claim 39, wherein the step of identifying oneor more of the previously categorized segments as relevant to theuncategorized segment further comprises the steps of: identifying aplurality of the previously categorized segments that are the mostsimilar to the uncategorized segment; determining the degree ofsimilarity between each of the plurality of previously categorizedsegments and each other of the plurality of previously categorizedsegments; for each pair of previously categorized segments of theplurality of previously categorized segments having greater than apredefined degree of similarity, eliminating one of the pair ofpreviously categorized segments from the plurality of previouslycategorized segments, wherein the previously categorized segment orsegments remaining after the step of eliminating are similar anddistinct previously categorized segments; and identifying one or more ofthe similar and distinct previously categorized segments as relevantpreviously categorized segments.
 42. A method as in claim 39, whereinthe step of selecting one or more subject matter categories furthercomprises selecting the most frequently occurring subject mattercategory or categories associated with the relevant previouslycategorized segments.
 43. A method as in claim 39, wherein theuncategorized segment has been acquired from a first data source and thepreviously categorized segment or segments have been acquired from asecond data source that is different than the first data source.
 44. Amethod as in claim 43, wherein: the data acquired from the first datasource are television or radio broadcast signals; and the data acquiredfrom the second data source are computer-readable data files.
 45. Amethod for determining whether a first set of information represented bya set of data of a first type is relevant to a second set of informationrepresented by a set of data of a second type, the first and second setsof information being different from each other, the method comprisingthe steps of: deriving a set of data of the second type from the set ofdata of the first type, the derived set of data of the second type alsobeing representative of the first set of information; determining thedegree of similarity between the set of data of the second typerepresenting the second set of information and the derived set of dataof the second type representing the first set of information; anddetermining whether the first set of information is relevant to thesecond set of information based upon the degree of similarity betweenthe set of data of the second type representing the second set ofinformation and the derived set of data of the second type representingthe first set of information.
 46. A method as in claim 45, wherein thefirst type of data is audiovisual data and the second type of data istext data.
 47. A method as in claim 46, wherein the step of determiningthe degree of similarity is accomplished using a relevance feedbackmethod.
 48. A method as in claim 45, wherein a plurality of sets ofinformation, each different from the other sets of the plurality of setsof information, are each represented by an associated set of data of thesecond type, the method enabling determination of which, if any, of theplurality of sets of information represented by a set of data of thesecond type are relevant to the first set of information represented bythe set of data of the first type, the method further comprising thesteps of: determining the degree of similarity between each set of dataof the second type representing one of the plurality of sets ofinformation and the derived set of data of the second type representingthe first set of information; identifying which, if any, of the sets ofdata of the second type representing one of the plurality of sets ofinformation have greater than a predefined degree of similarity to thederived set of data of the second type representing the first set ofinformation, the sets of data of the second type so identified beingtermed similar sets of data of the second type; determining the degreeof similarity between each similar set of data of the second type andeach other similar set of data of the second type; for each pair ofsimilar sets of data of the second type having greater than a predefineddegree of similarity, eliminating one of the pair of similar sets ofdata of the second type from the set of similar sets of data of thesecond type, wherein the set or sets of similar data of the second typeremaining after the step of eliminating are similar and distinct sets ofdata of the second type; and identifying the set or sets of informationcorresponding to one or more of the similar and distinct sets of data ofthe second type as relevant to the second set of information.
 49. Amethod as in claim 48, wherein the step of identifying the relevant setor sets of information further comprises identifying no more than apredetermined number of relevant sets of information, the predeterminednumber of relevant sets of information corresponding to the sets of dataof the second type having the greatest degree of similarity to thederived set of data of the second type.
 50. A method as in claim 45,wherein the first type of data is analog data and the second type ofdata is digital data.
 51. A method for identifying the boundaries ofsegments in a body of information, each segment comprising a contiguousrelated set of information in the body of information, wherein the bodyof information is represented by at least a set of text data and a setof video data, the method comprising the steps of: performing a coarsepartitioning method, the coarse partitioning method further comprisingthe steps of: identifying time-stamped markers in the set of text data;and determining approximate segment boundaries within the body ofinformation as the times of occurrence of the time-stamp markers; foreach approximate segment boundary, specifying a range of time thatincludes the time of occurrence of the approximate segment boundary;extracting subsets of video data from the set of video data that occurduring the specified ranges of time; performing a fine partitioningmethod to identify one or more breaks in the set of video data; andselecting the best break that occurs in each subset of video data, thetime of occurrence of the best break in each subset being designated asa boundary of a segment in the body of information.
 52. A method as inclaim 51, wherein the step of performing a fine partitioning methodfurther comprises identifying the best breaks using a process thatincludes scene break identification.
 53. A method as in claim 51,wherein the step of fine partitioning is performed on the entire set ofvideo data to identify all of the breaks in the set of video data.
 54. Amethod as in claim 51, wherein the step of fine partitioning isperformed only on the subsets of video data to identify only breaks thatoccur in the subsets.
 55. A method as in claim 51, wherein the bestbreak of each subset is determined according to the criteria of the finepartitioning method used.
 56. A method as in claim 51, wherein the bestbreak of each subset is the break occurring closest in time to the timeof occurrence of the segment boundary in the text data that correspondsto that subset.
 57. A method as in claim 51, wherein the body ofinformation is represented by a set of text data, a set of audio dataand a set of video data, the method further comprising the steps of:ascertaining a synchronization of the audio data and the video data; anddetermining the location of the segment boundaries in the set of audiodata using the previously determined location of the segment boundariesin the set of video data and the synchronization of the audio data andvideo data.
 58. A method for identifying the boundaries of segments in abody of information, each segment comprising a contiguous related set ofinformation in the body of information, wherein the body of informationis represented by a set of text data, a set of video data, and a set ofaudio data, the method comprising the steps of: performing a coarsepartitioning method, the coarse partitioning method further comprisingthe steps of: identifying time-stamped markers in the set of text data;and determining approximate segment boundaries within the body ofinformation as the times of occurrence of the time-stamp markers; foreach approximate segment boundary, specifying a range of time thatincludes the time of occurrence of the approximate segment boundary;extracting subsets of audio data from the set of audio data that occurduring the specified ranges of time; performing a fine partitioningmethod to identify one or more breaks in the set of audio data;selecting the best break that occurs in each subset of audio data, thetime of occurrence of the best break in each subset being designated asa boundary of a segment in the body of information; ascertaining asynchronization of the audio data and the video data; and determiningthe location of the segment boundaries in the set of video data usingthe previously determined location of the segment boundaries in the setof audio data and the synchronization of the audio data and video data.59. A method as in claim 58, wherein the step of performing finepartitioning further comprises identifying the best breaks using aprocess that includes pause recognition.
 60. A method as in claim 58,wherein the step of performing fine partitioning further comprisesidentifying the best breaks using a process that includes voicerecognition.
 61. A method as in claim 58, wherein the step of performingfine partitioning further comprises identifying the best breaks using aprocess that includes word recognition.
 62. A method as in claim 58,wherein the step of performing fine partitioning further comprisesidentifying the best breaks using a process that includes musicrecognition.
 63. A computer readable medium encoded with one or morecomputer programs for enabling acquisition and review of a body ofinformation, wherein the body of information includes a plurality ofsegments, each segment representing a defined set of information in thebody of information, comprising: instructions for acquiring datarepresenting the body of information; instructions for storing theacquired data; instructions for generating a display of a first segmentof the body of information from data that is part of the stored data;instructions for comparing data representing a segment of the body ofinformation to data representing a different segment of the body ofinformation to determine whether, according to one or more predeterminedcriteria, the compared segments are related; and instructions forgenerating a display of a portion of, or a representation of, a secondsegment of the body of information from data that is part of the storeddata, wherein the display of the portion or representation of the secondsegment is generated in response to the display of a first segment towhich the second segment is related.
 64. A computer readable medium asin claim 63, further comprising instructions for causing the display ofthe portion or representation of the second segment to occursubstantially coextensive in time with the display of the related firstsegment.
 65. A computer readable medium as in claim 63, wherein: theinstructions for acquiring data representing the body of informationfurther comprise instructions for acquiring audiovisual datarepresenting at least a portion of the body of information, wherein thefirst and second segments are represented by audiovisual data; and theinstructions for generating a display of a first segment of the body ofinformation further comprise instruction for generating an audiovisualdisplay of the first segment.
 66. A computer readable medium as in claim65, further comprising instructions for identifying the selection of asecond segment for which a portion or representation is being displayed,wherein selection of such second segment causes an audiovisual displayof the selected second segment to be produced.
 67. A computer readablemedium as in claim 63, wherein: the instructions for acquiring datarepresenting the body of information further comprise instructions foracquiring audiovisual data representing at least a portion of the bodyof information; the instructions for generating a display of a firstsegment of the body of information further comprise instructions forgenerating an audiovisual display of the first segment; and theinstructions for generating a display of a portion of, or arepresentation of, a second segment of the body of information furthercomprise instructions for generating a text display of the portion orrepresentation of the second segment.
 68. A computer readable medium asin claim 63, wherein: the instructions for generating a display of afirst segment of the body of information further comprise instructionsfor generating a display of the first segment on an analog displaydevice; and the instructions for generating a display of a portion of,or a representation of, a second segment of the body of informationfurther comprise instructions for generating a display of the portion orrepresentation of the second segment on a digital display device.
 69. Acomputer readable medium as in claim 63, wherein: the instructions forgenerating a display of the first segment on an analog display devicefurther comprise instructions for generating a display of the firstsegment on a television; and the instructions for generating a displayof the portion or representation of the second segment on a digitaldisplay device further comprise instructions for generating a display ofthe portion or representation of the second segment on a computerdisplay monitor.
 70. A computer readable medium as in claim 63, furthercomprising instructions for identifying the subject matter content of asegment of the body of information, wherein the instructions forcomparing further comprise instructions for determining the similarityof the subject matter content of a segment to the subject matter contentof a different segment, the predetermined criteria including apredefined degree of similarity with respect to which the relatedness ofthe compared segments is determined.
 71. A computer readable medium asin claim 70, wherein the instructions for determining the similarity ofthe subject matter of segments further comprise instructions forperforming a relevance feedback method.
 72. A computer readable mediumas in claim 63, wherein the instructions for acquiring data furthercomprise instructions for acquiring television broadcast signals.
 73. Acomputer readable medium as in claim 63, wherein the instructions foracquiring data further comprise instructions for acquiring radiobroadcast signals.
 74. A computer readable medium as in claim 63,wherein the instructions for acquiring data further compriseinstructions for acquiring computer-readable data files over a computernetwork from an information providing site that is part of that network.75. A computer readable medium as in claim 63, wherein the instructionsfor acquiring data further comprise: instructions for acquiringtelevision broadcast signals; and instructions for acquiringcomputer-readable data files over a computer network from an informationproviding site that is part of that network.
 76. A computer readablemedium as in claim 75, wherein: the first segment is represented by dataproduced from the television broadcast signals; and the second segmentis represented by data from the computer-readable data files.
 77. Acomputer readable medium as in claim 63, further comprising instructionsfor identifying an instruction from a user to begin displaying at leastsome of the body of information, wherein the display of a first segmentis begun in response to the user instruction.
 78. A computer readablemedium as in claim 63, wherein the first and second segments aredisplayed on physically separate display devices.
 79. A computerreadable medium as in claim 63, wherein the instructions for storing theacquired data, generating a display of a first segment of the body ofinformation, and generating a display of a portion of, or arepresentation of, a second segment of the body of information areexecuted by devices interconnected to a conventional computer bus thatenables the devices to communicate with each other such that the devicesdo not require wire communication over network communication lines tocommunicate with each other.
 80. A computer readable medium as in claim63, wherein at least some of the acquired data is digital data, theinstructions for acquiring data further comprising instructions foracquiring digital data.
 81. A computer readable medium as in claim 63,wherein at least some of the acquired data is analog data, theinstructions for acquiring data further comprising instructions foracquiring analog data.
 82. A computer readable medium encoded with oneor more computer programs for enabling categorization according tosubject matter of an uncategorized segment of a body of information thatincludes a plurality of segments, each segment representing a definedset of information in the body of information, one or more segmentshaving previously been categorized by identifying each of the one ormore segments with one or more subject matter categories, comprising:instructions for determining the degree of similarity between thesubject matter content of the uncategorized segment and the subjectmatter content of each of the previously categorized segments;instructions for identifying one or more of the previously categorizedsegments as relevant to the uncategorized segment based upon thedetermined degrees of similarity of subject matter content between theuncategorized segment and the previously categorized segments; andinstructions for selecting one or more subject matter categories withwhich to identify the uncategorized segment based upon the subjectmatter categories used to identify the relevant previously categorizedsegments.
 83. A computer readable medium as in claim 82, wherein theinstructions for determining the degree of similarity further compriseinstructions for performing a relevance feedback method.
 84. A computerreadable medium as in claim 82, wherein the instructions for identifyingone or more of the previously categorized segments as relevant to theuncategorized segment further comprise: instructions for identifying aplurality of the previously categorized segments that are the mostsimilar to the uncategorized segment; instructions for determining thedegree of similarity between each of the plurality of previouslycategorized segments and each other of the plurality of previouslycategorized segments; instructions for eliminating, for each pair ofpreviously categorized segments of the plurality of previouslycategorized segments having greater than a predefined degree ofsimilarity, one of the pair of previously categorized segments from theplurality of previously categorized segments, wherein the remainingpreviously categorized segment or segments are similar and distinctpreviously categorized segments; and instructions for identifying one ormore of the similar and distinct previously categorized segments asrelevant previously categorized segments.
 85. A computer readable mediumas in claim 82, wherein the instructions for selecting one or moresubject matter categories further comprise instructions for selectingthe most frequently occurring subject matter category or categoriesassociated with the relevant previously categorized segments.
 86. Acomputer readable medium as in claim 82, wherein the uncategorizedsegment has been acquired from a first data source and the previouslycategorized segment or segments have been acquired from a second datasource that is different than the first data source.
 87. A computerreadable medium as in claim 86, wherein: the data acquired from thefirst data source are television or radio broadcast signals; and thedata acquired from the second data source are computer-readable datafiles.
 88. A computer readable medium encoded with one or more computerprograms for enabling determination of whether a first set ofinformation represented by a set of data of a first type is relevant toa second set of information represented by a set of data of a secondtype, the first and second sets of information being different from eachother, comprising: instructions for deriving a set of data of the secondtype from the set of data of the first type, the derived set of data ofthe second type also being representative of the first set ofinformation; instructions for determining the degree of similaritybetween the set of data of the second type representing the second setof information and the derived set of data of the second typerepresenting the first set of information; and instructions fordetermining whether the first set of information is relevant to thesecond set of information based upon the degree of similarity betweenthe set of data of the second type representing the second set ofinformation and the derived set of data of the second type representingthe first set of information.
 89. A computer readable medium as in claim88, wherein the first type of data is audiovisual data and the secondtype of data is text data.
 90. A computer readable medium as in claim89, wherein the instructions for determining the degree of similarityfurther comprise instructions for performing a relevance feedbackmethod.
 91. A computer readable medium as in claim 88, wherein aplurality of sets of information, each different from the other sets ofthe plurality of sets of information, are each represented by anassociated set of data of the second type, the one or more computerprograms enabling determination of which, if any, of the plurality ofsets of information represented by a set of data of the second type arerelevant to the first set of information represented by the set of dataof the first type, the one or more computer programs further comprising:instructions for determining the degree of similarity between each setof data of the second type representing one of the plurality of sets ofinformation and the derived set of data of the second type representingthe first set of information; instructions for identifying which, ifany, of the sets of data of the second type representing one of theplurality of sets of information have greater than a predefined degreeof similarity to the derived set of data of the second type representingthe first set of information, the sets of data of the second type soidentified being termed similar sets of data of the second type;instructions for determining the degree of similarity between eachsimilar set of data of the second type and each other similar set ofdata of the second type; instructions for eliminating, for each pair ofsimilar sets of data of the second type having greater than a predefineddegree of similarity, one of the pair of similar sets of data of thesecond type from the set of similar sets of data of the second type,wherein the remaining set or sets of similar data of the second type aresimilar and distinct sets of data of the second type; and instructionsfor identifying the set or sets of information corresponding to one ormore of the similar and distinct sets of data of the second type asrelevant to the second set of information.
 92. A computer readablemedium as in claim 91, wherein the instructions for identifying therelevant set or sets of information further comprise instructions foridentifying no more than a predetermined number of relevant sets ofinformation, the predetermined number of relevant sets of informationcorresponding to the sets of data of the second type having the greatestdegree of similarity to the derived set of data of the second type. 93.A computer readable medium as in claim 88, wherein the first type ofdata is analog data and the second type of data is digital data.
 94. Acomputer readable medium encoded with one or more computer programs forenabling identification of the boundaries of segments in a body ofinformation, each segment comprising a contiguous related set ofinformation in the body of information, wherein the body of informationis represented by at least a set of text data and a set of video data,comprising: instructions for performing a coarse partitioning method,the coarse partitioning instructions further comprising: instructionsfor identifying time-stamped markers in the set of text data; andinstructions for determining approximate segment boundaries within thebody of information as the times of occurrence of the time-stampmarkers; instructions for specifying, for each approximate segmentboundary, a range of time that includes the time of occurrence of theapproximate segment boundary; instructions for extracting subsets ofvideo data from the set of video data that occur during the specifiedranges of time; instructions for performing a fine partitioning methodto identify one or more breaks in the set of video data; andinstructions for selecting the best break that occurs in each subset ofvideo data, the time of occurrence of the best break in each subsetbeing designated as a boundary of a segment in the body of information.95. A computer readable medium as in claim 94, wherein the instructionsfor performing a fine partitioning method further comprise instructionsfor identifying the best breaks using a process that includes scenebreak identification.
 96. A computer readable medium as in claim 94,wherein the fine partitioning method is performed on the entire set ofvideo data to identify all of the breaks in the set of video data.
 97. Acomputer readable medium as in claim 94, wherein the fine partitioningmethod is performed only on the subsets of video data to identify onlybreaks that occur in the subsets.
 98. A computer readable medium as inclaim 94, wherein the best break of each subset is determined accordingto the criteria of the fine partitioning method used.
 99. A computerreadable medium as in claim 94, wherein the best break of each subset isthe break occurring closest in time to the time of occurrence of thesegment boundary in the text data that corresponds to that subset. 100.A computer readable medium as in claim 94, wherein the body ofinformation is represented by a set of text data, a set of audio dataand a set of video data, the one or more computer programs furthercomprising: instructions for ascertaining a synchronization of the audiodata and the video data; and instructions for determining the locationof the segment boundaries in the set of audio data using the previouslydetermined location of the segment boundaries in the set of video dataand the synchronization of the audio data and video data.
 101. A systemfor categorizing according to subject matter an uncategorized segment ofa body of information that includes a plurality of segments, eachsegment representing a defined set of information in the body ofinformation, one or more segments of the body of information havingpreviously been categorized by identifying each of the one or moresegments with one or more subject matter categories, the systemcomprising: means for determining the degree of similarity between thesubject matter content of the uncategorized segment and the subjectmatter content of each of the previously categorized segments; means foridentifying one or more of the previously categorized segments asrelevant to the uncategorized segment based upon the determined degreesof similarity of subject matter content between the uncategorizedsegment and the previously categorized segments; and means for selectingone or more subject matter categories with which to identify theuncategorized segment based upon the subject matter categories used toidentify the relevant previously categorized segments.
 102. A system asin claim 101, wherein the means for determining the degree of similarityfurther comprises means for performing a relevance feedback method. 103.A system as in claim 101, wherein the means for identifying one or moreof the previously categorized segments as relevant to the uncategorizedsegment further comprises: means for identifying a plurality of thepreviously categorized segments that are the most similar to theuncategorized segment; means for determining the degree of similaritybetween each of the plurality of previously categorized segments andeach other of the plurality of previously categorized segments; meansfor eliminating, for each pair of previously categorized segments of theplurality of previously categorized segments having greater than apredefined degree of similarity, one of the pair of previouslycategorized segments from the plurality of previously categorizedsegments, wherein the remaining previously categorized segment orsegments are similar and distinct previously categorized segments; andmeans for identifying one or more of the similar and distinct previouslycategorized segments as relevant previously categorized segments.
 104. Asystem as in claim 101, wherein the means for selecting one or moresubject matter categories further comprises means for selecting the mostfrequently occurring subject matter category or categories associatedwith the relevant previously categorized segments.
 105. A system as inclaim 101, wherein the uncategorized segment has been acquired from afirst data source and the previously categorized segment or segmentshave been acquired from a second data source that is different than thefirst data source.
 106. A system as in claim 105, wherein: the dataacquired from the first data source are television or radio broadcastsignals; and the data acquired from the second data source arecomputer-readable data files.
 107. A system for determining whether afirst set of information represented by a set of data of a first type isrelevant to a second set of information represented by a set of data ofa second type, the first and second sets of information being differentfrom each other, the system comprising: means for deriving a set of dataof the second type from the set of data of the first type, the derivedset of data of the second type also being representative of the firstset of information; means for determining the degree of similaritybetween the set of data of the second type representing the second setof information and the derived set of data of the second typerepresenting the first set of information; and means for determiningwhether the first set of information is relevant to the second set ofinformation based upon the degree of similarity between the set of dataof the second type representing the second set of information and thederived set of data of the second type representing the first set ofinformation.
 108. A system as in claim 107, wherein the first type ofdata is audiovisual data and the second type of data is text data. 109.A system as in claim 108, wherein the means for determining the degreeof similarity further comprises means for performing a relevancefeedback method.
 110. A system as in claim 107, wherein a plurality ofsets of information, each different from the other sets of the pluralityof sets of information, are each represented by an associated set ofdata of the second type, the system enabling determination of which, ifany, of the plurality of sets of information represented by a set ofdata of the second type are relevant to the first set of informationrepresented by the set of data of the first type, the system furthercomprising: means for determining the degree of similarity between eachset of data of the second type representing one of the plurality of setsof information and the derived set of data of the second typerepresenting the first set of information; means for identifying which,if any, of the sets of data of the second type representing one of theplurality of sets of information have greater than a predefined degreeof similarity to the derived set of data of the second type representingthe first set of information, the sets of data of the second type soidentified being termed similar sets of data of the second type; meansfor determining the degree of similarity between each similar set ofdata of the second type and each other similar set of data of the secondtype; means for eliminating, for each pair of similar sets of data ofthe second type having greater than a predefined degree of similarity,one of the pair of similar sets of data of the second type from the setof similar sets of data of the second type, wherein the remaining set orsets of similar data of the second type are similar and distinct sets ofdata of the second type; and means for identifying the set or sets ofinformation corresponding to one or more of the similar and distinctsets of data of the second type as relevant to the second set ofinformation.
 111. A system as in claim 110, wherein the means foridentifying the relevant set or sets of information further comprisesmeans for identifying no more than a predetermined number of relevantsets of information, the predetermined number of relevant sets ofinformation corresponding to the sets of data of the second type havingthe greatest degree of similarity to the derived set of data of thesecond type.
 112. A system as in claim 107, wherein the first type ofdata is analog data and the second type of data is digital data.
 113. Acomputer readable medium encoded with one or more computer programs foridentifying the boundaries of segments in a body of information, eachsegment comprising a contiguous related set of information in the bodyof information, wherein the body of information is represented by a setof text data, a set of video data, and a set of audio data, comprising:instructions for performing a coarse partitioning method, theinstructions for performing a coarse partitioning method furthercomprising: instructions for identifying time-stamped markers in the setof text data; and instructions for determining approximate segmentboundaries within the body of information as the times of occurrence ofthe time-stamp markers; instructions for specifying, for eachapproximate segment boundary, a range of time that includes the time ofoccurrence of the approximate segment boundary; instructions forextracting subsets of audio data from the set of audio data that occurduring the specified ranges of time; instructions for performing a finepartitioning method to identify one or more breaks in the set of audiodata; instructions for selecting the best break that occurs in eachsubset of audio data, the time of occurrence of the best break in eachsubset being designated as a boundary of a segment in the body ofinformation; instructions for ascertaining a synchronization of theaudio data and the video data; and instructions for determining thelocation of the segment boundaries in the set of video data using thepreviously determined location of the segment boundaries in the set ofaudio data and the synchronization of the audio data and video data.114. A computer readable medium as in claim 113, wherein theinstructions for performing fine partitioning further compriseinstructions for identifying the best breaks using a process thatincludes pause recognition.
 115. A computer readable medium as in claim113, wherein the instructions for performing fine partitioning furthercomprise instructions for identifying the best breaks using a processthat includes voice recognition.
 116. A computer readable medium as inclaim 113, wherein the instructions for performing fine partitioningfurther comprise instructions for identifying the best breaks using aprocess that includes word recognition.
 117. A computer readable mediumas in claim 113, wherein the instructions for performing finepartitioning further comprise instructions for identifying the bestbreaks using a process that includes music recognition.
 118. A systemfor identifying the boundaries of segments in a body of information,each segment comprising a contiguous related set of information in thebody of information, wherein the body of information is represented byat least a set of text data and a set of video data, the systemcomprising: means for performing a coarse partitioning method, the meansfor performing a coarse partitioning method further comprising: meansfor identifying time-stamped markers in the set of text data; and meansfor determining approximate segment boundaries within the body ofinformation as the times of occurrence of the time-stamp markers; meansfor specifying, for each approximate segment boundary, a range of timethat includes the time of occurrence of the approximate segmentboundary; means for extracting subsets of video data from the set ofvideo data that occur during the specified ranges of time; means forperforming a fine partitioning method to identify one or more breaks inthe set of video data; and means for selecting the best break thatoccurs in each subset of video data, the time of occurrence of the bestbreak in each subset being designated as a boundary of a segment in thebody of information.
 119. A system as in claim 118, wherein the meansfor performing a fine partitioning method further comprises means foridentifying the best breaks using a process that includes scene breakidentification.
 120. A system as in claim 118, wherein the finepartitioning method is performed on the entire set of video data toidentify all of the breaks in the set of video data.
 121. A system as inclaim 118, wherein the fine partitioning method is performed only on thesubsets of video data to identify only breaks that occur in the subsets.122. A system as in claim 118, wherein the best break of each subset isdetermined according to the criteria of the fine partitioning methodused.
 123. A system as in claim 118, wherein the best break of eachsubset is the break occurring closest in time to the time of occurrenceof the segment boundary in the text data that corresponds to thatsubset.
 124. A system as in claim 118, wherein the body of informationis represented by a set of text data, a set of audio data and a set ofvideo data, the system further comprising: means for ascertaining asynchronization of the audio data and the video data; and means fordetermining the location of the segment boundaries in the set of audiodata using the previously determined location of the segment boundariesin the set of video data and the synchronization of the audio data andvideo data.
 125. A system for identifying the boundaries of segments ina body of information, each segment comprising a contiguous related setof information in the body of information, wherein the body ofinformation is represented by a set of text data, a set of video data,and a set of audio data, the system comprising: means for performing acoarse partitioning method, the means for performing a coarsepartitioning method further comprising: means for identifyingtime-stamped markers in the set of text data; and means for determiningapproximate segment boundaries within the body of information as thetimes of occurrence of the time-stamp markers; means for specifying, foreach approximate segment boundary, a range of time that includes thetime of occurrence of the approximate segment boundary; means forextracting subsets of audio data from the set of audio data that occurduring the specified ranges of time; means for performing a finepartitioning method to identify one or more breaks in the set of audiodata; means for selecting the best break that occurs in each subset ofaudio data, the time of occurrence of the best break in each subsetbeing designated as a boundary of a segment in the body of information;means for ascertaining a synchronization of the audio data and the videodata; and means for determining the location of the segment boundariesin the set of video data using the previously determined location of thesegment boundaries in the set of audio data and the synchronization ofthe audio data and video data.
 126. A system as in claim 125, whereinthe means for performing a fine partitioning method further comprisesmeans for identifying the best breaks using a process that includespause recognition.
 127. A system as in claim 125, wherein the means forperforming a fine partitioning method further comprises means foridentifying the best breaks using a process that includes voicerecognition.
 128. A system as in claim 125, wherein the means forperforming a fine partitioning method further comprises means foridentifying the best breaks using a process that includes wordrecognition.
 129. A system as in claim 125, wherein the means forperforming a fine partitioning method further comprises means foridentifying the best breaks using a process that includes musicrecognition.